Imaging system and imaging method

ABSTRACT

An imaging system includes an imaging optical system, an imaging device, an actuator, and control circuitry. The actuator changes a relative position of a plurality of pixel cells and an image of a subject. The pixel cells have variable sensitivity, and include a photoelectric converter and a charge accumulation region. The control circuitry sets the relative position to a first position, and also sets the sensitivity of each pixel cell to a first sensitivity. A first signal charge obtained at the photoelectric converter is accumulated in the charge accumulation region. The relative position is set to a second position different from the first position, and also the sensitivity of each pixel cell is set to a second sensitivity different from the first sensitivity. A second signal charge obtained at the photoelectric converter is accumulated in the charge accumulation region in addition to the first signal charge.

BACKGROUND 1. Technical Field

The present disclosure relates to an imaging system and imaging method.

2. Description of the Related Art

So-called image processing, where computation processing is performed onnumerical value data made up of a two-dimensionally arrayed numericalvalue group corresponding to image luminance or the like, is widelyperformed for enhancement or extraction of image features, removal ofunnecessary noise, image recognition, and so forth.

A technique often used in this image processing is processing called“convolution”. Convolution is computation using numerical value data ofan original image, and a coefficient group called a filter that isprepared separately from the numerical value data of the original image.

On the other hand, Japanese Unexamined Patent Application PublicationNo. 2010-87850 discloses a technology where output distribution of animage sensor can be made to yield desired low-pass filter properties, bymoving the position of an image sensor while exposing.

Japanese Examined Patent Application Publication No. 02-51316 disclosestechnology where a solid-state imaging device is relatively vibrated asto an incident optical image, thereby effectively widening the aperturearea.

SUMMARY

One non-limiting and exemplary embodiment provides an imaging systemwhereby the computation amount of convolution processing can be reduced.

In one general aspect, the techniques disclosed here feature an imagingsystem including an imaging optical system that images an image of asubject, an imaging device including a plurality of pixel cells, anactuator that changes a relative position of the plurality of pixelcells and the image of the subject, and control circuitry that controlsthe imaging device and the actuator. The plurality of pixel cells eachhave variable sensitivity, and the plurality of pixel cells each includea photoelectric converter that converts light of the image of thesubject into a signal charge, and a charge accumulation region thataccumulates the signal charge obtained at the photoelectric converter.The control circuitry sets the relative position to a first position,and also sets the sensitivity of each of the plurality of pixel cells toa first sensitivity, to cause a first signal charge obtained at thephotoelectric converter to be accumulated in the charge accumulationregion in each of the plurality of pixel cells, and sets the relativeposition to a second position that is different from the first position,and also sets the sensitivity of each of the plurality of pixel cells toa second sensitivity that is different from the first sensitivity, tocause a second signal charge obtained at the photoelectric converter tobe accumulated in the charge accumulation region in each of theplurality of pixel cells in addition to the first signal charge.

An imaging system according to an aspect of the present disclosureenables the computation amount of convolution processing to be reduced.

It should be noted that general or specific embodiments may beimplemented as a device, an apparatus, a system, a method, an integratedcircuit, a computer program, a storage medium, or any selectivecombination thereof.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an example of a filter for convolutionprocessing;

FIG. 2 is a diagram illustrating an example of an original image;

FIG. 3 is a diagram illustrating numerical value data of an originalimage;

FIG. 4 is a diagram illustrating numerical value data after convolutionprocessing;

FIG. 5 is a diagram illustrating an image corresponding to numericalvalue data after convolution processing;

FIG. 6 is a diagram illustrating an example of convolution processing;

FIG. 7 is a conceptual configuration diagram illustrating an example ofan imaging system according to a first embodiment;

FIG. 8 is a schematic diagram illustrating an exemplary circuitconfiguration of an imaging device according to the first embodiment;

FIG. 9 is a configuration diagram illustrating an example of a layeredimaging device with variable sensitivity, according to the firstembodiment;

FIG. 10 is a diagram illustrating an example of bias voltage dependencyof the sensitivity of the imaging device according to the firstembodiment;

FIG. 11 is a diagram illustrating an example of a relative positionaccording to the first embodiment;

FIG. 12 is a flowchart illustrating imaging procedures according to thefirst embodiment;

FIG. 13 is a flowchart illustrating imaging procedures according to thefirst embodiment;

FIG. 14 is a flowchart illustrating imaging procedures according to thefirst embodiment;

FIG. 15 is a timing chart illustrating an example of the flow of imagingprocessing according to the first embodiment;

FIG. 16 is a diagram illustrating an example of the flow of imagingprocedures according to the first embodiment;

FIG. 17 is a diagram for describing imaging processing according to asecond embodiment;

FIG. 18 is a diagram for describing imaging processing according to asecond embodiment;

FIG. 19 is a diagram illustrating an example of change in relativeposition, according to a third embodiment;

FIG. 20 is a diagram illustrating an example of change in relativeposition, according to the third embodiment;

FIG. 21 is a diagram illustrating an example of change in relativeposition, according to the third embodiment;

FIG. 22 is a diagram illustrating an example of pixel configurationsaccording to a fourth embodiment;

FIG. 23 is a flowchart illustrating imaging procedures according to thefourth embodiment; and

FIG. 24 is a diagram illustrating an example of the flow of imagingprocessing according to a fourth embodiment.

DETAILED DESCRIPTION Underlying Knowledge Forming Basis of the PresentDisclosure

In a case of performing convolution processing using software, theamount of computation increases in proportion to the increase in thenumber of pixels. Convolution processing is widely used in imagerecognition including so-called deep learning. Specifically, there arecases in processing such as deep learning and the like where convolutionprocessing is performed on one image using more than 100 filters. Inthis case, increase in the number of pixels causes the amount ofcalculations to become massive. This problem is relatively small in acase where processing is performed taking time with a server that hashigh computation capabilities, but reduction in the amount ofcomputations is strongly demanded for performing convolution withdevices that have limited computation capabilities, such as mobiledevices and so forth.

Accordingly, the present disclosure provides an imaging system andimaging method that enables the amount of computations for convolutionprocessing to be reduced.

An imaging system according to an aspect of the present disclosureincludes an imaging optical system that images an image of a subject, animaging device including a plurality of pixel cells, an actuator thatchanges a relative position of the plurality of pixel cells and theimage of the subject, and control circuitry that controls the imagingdevice and the actuator. The plurality of pixel cells each have variablesensitivity, and the plurality of pixel cells each include aphotoelectric converter that converts light of the image of the subjectinto a signal charge, and a charge accumulation region that accumulatesthe signal charge obtained at the photoelectric converter. The controlcircuitry sets the relative position to a first position, and also setsthe sensitivity of each of the plurality of pixel cells to a firstsensitivity, to cause a first signal charge obtained at thephotoelectric converter to be accumulated in the charge accumulationregion in each of the plurality of pixel cells. The control circuitryfurther sets the relative position to a second position that isdifferent from the first position, and also sets the sensitivity of eachof the plurality of pixel cells to a second sensitivity that isdifferent from the first sensitivity, to cause a second signal chargeobtained at the photoelectric converter to be accumulated in the chargeaccumulation region in each of the plurality of pixel cells in additionto the first signal charge.

According to this configuration, the imaging system can obtain a pieceof image data, in which signal charges obtained at different relativepositions and different sensitivities have been accumulated, whilechanging the relative position between the plurality of pixel cells andthe image of the subject, and changing the sensitivity of the pluralityof pixel cells. Accordingly, at least part of convolution computationprocessing can be performed at the imaging device, so the amount ofcomputation of convolution processing can be reduced.

The control circuitry may synchronously perform setting of the relativeposition to the first position, and setting of the sensitivity of eachof the plurality of pixel cells to the first sensitivity. The controlcircuitry may also synchronously perform setting of the relativeposition to the second position, and setting of the sensitivity of eachof the plurality of pixel cells to the second sensitivity.

The control circuitry may perform the setting to the second position bychanging the relative position from the first position to the secondposition by an integer multiple of a center-to-center distance betweentwo adjacent pixel cells out of the plurality of pixel cells.Accordingly, the imaging system can improve replicability of convolutioncomputation.

An imaging system according to another aspect of the present disclosureincludes an imaging optical system that images an image of a subject, animaging device including a plurality of pixel cells, an actuator thatchanges a relative position of the plurality of pixel cells and theimage of the subject, and control circuitry that controls the imagingdevice and the actuator. The plurality of pixel cells each have variablesensitivity, and the plurality of pixel cells each include aphotoelectric converter that converts light of the image of the subjectinto a signal charge, and a charge accumulation region that accumulatesthe signal charge obtained at the photoelectric converter. During oneexposure period, the control circuitry changes the relative positionfrom a first position to a second position that is different from thefirst position, and changes the sensitivity of each of the plurality ofpixel cells from a first sensitivity to a second sensitivity that isdifferent from the first sensitivity, to cause a third signal chargeobtained at the photoelectric converter to be accumulated at the chargeaccumulation region, in each of the plurality of pixel cells.

According to this configuration, the imaging system can obtain a pieceof image data, in which signal charges obtained at different relativepositions and different sensitivities have been accumulated, whilechanging the relative position between the plurality of pixel cells andthe image of the subject, and changing the sensitivity of the pluralityof pixel cells. Accordingly, at least part of convolution computationprocessing can be performed at the imaging device, so the amount ofcomputation of convolution processing can be reduced.

The control circuitry may change the relative position from the firstposition to the second position in a continuous manner.

According to this configuration, the imaging system can easily realizechanging of the relative position between the plurality of pixel cellsand the image of the subject, and increase in shooting time due to thischanging can be suppressed.

The control circuitry may change the sensitivity of each of theplurality of pixel cells from the first sensitivity to the secondsensitivity in a continuous manner. Accordingly, the imaging system canimprove replicability of convolution computation.

The control circuitry may perform the changing of the relative positionand the changing of the sensitivity of each of the plurality of pixelcells in a synchronous manner.

The plurality of pixel cells may be laid out two-dimensionally in a rowdirection and a column direction.

The control circuitry may perform the setting or the changing of thesensitivity of each of the plurality of pixel cells all at once.

According to this configuration, the mechanism of the imaging system forchanging sensitivity can be simplified.

Each of the plurality of pixel cells may include a first sub-pixel and asecond sub-pixel. The first sub-pixel of each of the plurality of pixelcells may include the photoelectric converter and the chargeaccumulation region. The sensitivity of each of the plurality of pixelcells may be the sensitivity of the first sub-pixel. The secondsub-pixel of each of the plurality of pixel cells may include a secondphotoelectric converter and a second charge accumulation region. Thecontrol circuitry may set or change sensitivity of the second sub-pixelindependently from the sensitivity of the first sub-pixel in each of theplurality of pixel cells.

According to this configuration, the imaging system can generate aplurality of images obtained by different convolution computation at thesame time.

The photoelectric converter may include a pixel electrode connected tothe charge accumulation region, an opposing electrode that transmitslight, and a photoelectric conversion layer disposed between the pixelelectrode and the opposing electrode.

The imaging device may further include a voltage applying circuit thatapplies voltage across the pixel electrode and the opposing electrodeincluded in the photoelectric converter, in each of the plurality ofpixel cells. The control circuitry may change the sensitivity of each ofthe plurality of pixel cells by changing the voltage that the voltageapplying circuit applies.

According to this configuration, the imaging system can change thesensitivity of the pixel cells by changing the voltage applied to thephotoelectric converter.

An imaging method according to an aspect of the present disclosure is animaging method in an imaging system. The imaging system includes animaging optical system that images an image of a subject, an imagingdevice including a plurality of pixel cells, an actuator that changes arelative position of the plurality of pixel cells and the image of thesubject, and control circuitry that controls the imaging device and theactuator. The plurality of pixel cells each have variable sensitivity,and the plurality of pixel cells each include a photoelectric converterthat converts light of the image of the subject into a signal charge,and a charge accumulation region that accumulates the signal chargeobtained at the photoelectric converter. The imaging method includessetting the relative position to a first position, setting thesensitivity of each of the plurality of pixel cells to a firstsensitivity, causing a first signal charge obtained at the photoelectricconverter to be accumulated in the charge accumulation region in each ofthe plurality of pixel cells in a state where the relative position isset to the first position and the sensitivity of each of the pluralityof pixel cells set to the first sensitivity, setting the relativeposition to a second position that is different from the first position,setting the sensitivity of each of the plurality of pixel cells to asecond sensitivity that is different from the first sensitivity, andcausing a second signal charge obtained at the photoelectric converterto be accumulated in the charge accumulation region in each of theplurality of pixel cells in addition to the first signal charge, in astate where the relative position is set to the second position and thesensitivity of each of the plurality of pixel cells set to the secondsensitivity.

According to this configuration, the imaging method can obtain a pieceof image data, in which signal charges obtained at different relativepositions and different sensitivities have been accumulated, whilechanging the relative position between the plurality of pixel cells andthe image of the subject, and changing the sensitivity of the pluralityof pixel cells. Accordingly, at least part of convolution computationprocessing can be performed at the imaging device, so the amount ofcomputation of convolution processing can be reduced.

The setting to the second position may be performed by changing therelative position from the first position to the second position by aninteger multiple of a center-to-center distance between two adjacentpixel cells out of the plurality of pixel cells. Accordingly, theimaging method can improve replicability of convolution processing.

The imaging system may further have a mechanical shutter that shieldsimaging device from light. The setting to the second position may beperformed by changing the relative position from the first position tothe second position in a state where the imaging device is shielded fromlight by the mechanical shutter.

According to this configuration, the imaging method can improvereplicability of convolution computation, since exposure is notperformed while changing the relative position.

The setting to the second position may be performed by changing therelative position from the first position to the second position in astate where the sensitivity of each of the plurality of pixel cells setto zero.

According to this configuration, the imaging method can improvereplicability of convolution computation, since an arrangement can bemade where exposure is not performed while changing the relativeposition.

The imaging method may further include: performing an N count (where Nis an integer of 2 or greater) of settings of the relative position,including the setting to the first position and the setting to thesecond position, in which the relative position is set to an i'th (wherei is an integer of 1 to N) position in an i'th setting of the relativeposition; performing an N count of settings of the sensitivity,including the setting to the first sensitivity and the setting to thesecond sensitivity, in which the sensitivity of each of the plurality ofpixel cells is set to an i'th sensitivity in an i'th setting of thesensitivity; performing an N count of accumulations, including theaccumulation of the first signal charge and the accumulation of thesecond signal charge, in which the i'th signal charge obtained at thephotoelectric converter is accumulated at the charge accumulation regionat each of the plurality of pixel cells in a state where the relativeposition is set to the i'th position and the sensitivity of each of theplurality of pixel cells is set to the i'th sensitivity at the i'thaccumulation; and obtaining a piece of image data using one or morepieces of image data obtained by the N count of the accumulations. Thepiece of image data may be equivalent to a piece of image dataobtainable by predetermined first convolution processing. The i'thposition may correspond to a position of an i'th coefficient included inan N count of coefficients out of a plurality of coefficients in thefirst convolution processing. The i'th sensitivity may correspond to thevalue of the i'th coefficient.

According to this configuration, at least part of convolutioncomputation processing can be performed at the imaging device, so theamount of computation of convolution processing can be reduced.

The one or more pieces of image data may include a first piece of imagedata and a second piece of image data. The N may be an integer of 3 orgreater. The N count of coefficients may be made up of an M (wherein Mis an integer of 1 or greater but smaller than N) count of coefficientshaving a positive value, and an (N−M) count of coefficients having anegative value. The first piece of image data may correspond to a sumvalue of signal charges accumulated in the charge accumulation region ofeach of the plurality of pixel cells, by the M count of accumulationscorresponding to the M count of coefficients. The second piece of imagedata may correspond to a sum value of signal charges accumulated in thecharge accumulation region of each of the plurality of pixel cells, byan (N−M) count of accumulations corresponding to the (N−M) count ofnegative coefficients. The piece of image data equivalent to the pieceof image data obtained by the first convolution processing may beobtained by subtracting the second piece of image data from the firstpiece of image data.

According to this configuration, convolution computation processingincluding negative coefficients can be realized by this imaging method.

The imaging method may further include: performing an N count (where Nis an integer of 4 or greater) of settings of the relative position,including the setting to the first position and the setting to thesecond position, in which the relative position is set to an i'th (wherei is an integer of 1 to N) position in an i'th setting of the relativeposition; performing an N count of settings of the sensitivity,including the setting to the first sensitivity and the setting to thesecond sensitivity, in which the sensitivity of each of the plurality ofpixel cells is set to an i'th sensitivity in an i'th setting of thesensitivity; performing an N count of accumulations, including theaccumulation of the first signal charge and the accumulation of thesecond signal charge, in which the i'th signal charge obtained at thephotoelectric converter is accumulated at the charge accumulation regionat each of the plurality of pixel cells in a state where the relativeposition is set to the i'th position and the sensitivity of each of theplurality of pixel cells is set to the i'th sensitivity at the i'thaccumulation; performing a setting to a sensitivity corresponding to asum value of an offset value and one of a plurality of coefficients ofpredetermined first convolution processing, in each of an M (wherein Mis an integer of 2 or greater but smaller than N) counts of settings,out of the N counts of settings of the sensitivity; performing a settingto sensitivity corresponding to the offset value in each of the (N−M)count of settings, out of the N counts of settings of the sensitivity;obtaining a first piece of image data by an M count of accumulationscorresponding to the M count of settings of the sensitivity, the firstpiece of image data being equivalent to a piece of image data obtainableby convolution processing using a plurality of coefficients obtainableby adding an offset value to all coefficients of the first convolutionprocessing; obtaining a second piece of image data by an (N−M) count ofaccumulations corresponding to the (N−M) count of settings of thesensitivity, the second piece of image data being equivalent a piece ofimage data obtainable by convolution processing using the offset valueas all coefficients; and obtaining a piece of image data by subtractingthe second piece of image data from the first piece of image data. Thepiece of image data may be equivalent to a piece of image dataobtainable by the first convolution processing.

According to this configuration, convolution computation processingincluding negative coefficients can be realized by this imaging method.In a case where a plurality of images subjected to different convolutionprocessing from each other are obtained, the second piece of image datacan be used in common, so shooting time can be reduced.

The value of the N count of coefficients used for the N count ofsettings of the sensitivity may not be 0. Accordingly, the shooting timefor the imaging method can be reduced.

The imaging method may further include: obtaining a piece of image dataequivalent to a piece of image data obtainable by second convolutionprocessing that is different from the first convolution processing. Eachof the plurality of pixel cells may include a first sub-pixel and asecond sub-pixel. The first sub-pixel of each of the plurality of pixelcells may include the photoelectric converter and the chargeaccumulation region. The sensitivity of each of the plurality of pixelcells may be sensitivity of the first sub-pixel. The second sub-pixel ofeach of the plurality of pixel cells may include a second photoelectricconverter and a second charge accumulation region. In each of the Ncount of settings of the sensitivity, sensitivity of the secondsub-pixel in each of the plurality of pixel cells may be further set. Inat least one of the N count of settings of the sensitivity, a settingvalue of the sensitivity of the first sub-pixel may be different from asetting value of the sensitivity of the second sub-pixel. In each of theN count of accumulations, a signal charge obtained at the secondphotoelectric converter of the second sub-pixel may be accumulated inthe second charge accumulation region, in each of the plurality of pixelcells. The piece of image data equivalent to the piece of image dataobtainable by the first convolution processing may be obtained using oneor more pieces of image data obtained by the plurality of firstsub-pixels by the N count of accumulations. The piece of image dataequivalent to the piece of image data obtainable by the secondconvolution processing may be obtained using one or more other imagesobtained by the plurality of second sub-pixels by the N count ofaccumulations.

Accordingly, a plurality of images obtained by different convolutioncomputation can be obtained by this imaging method.

The imaging method may further include: performing an N count (where Nis an integer of 2 or greater) of settings of the relative position,including the setting to the first position and the setting to thesecond position, in which the relative position is set to an i'th (wherei is an integer of 1 to N) position in an i'th setting of the relativeposition; performing an N count of settings of the sensitivity,including the setting to the first sensitivity and the setting to thesecond sensitivity, in which the sensitivity of each of the plurality ofpixel cells is set to an i'th sensitivity in an i'th setting of thesensitivity, and the exposure time is set to an i'th exposure time;performing an N count of accumulations, including the accumulation ofthe first signal charge and the accumulation of the second signalcharge, in which the i'th signal charge obtained at the photoelectricconverter in the i'th exposure time is accumulated at the chargeaccumulation region at each of the plurality of pixel cells in a statewhere the relative position is set to the i'th position and thesensitivity of each of the plurality of pixel cells is set to the i'thsensitivity at the i'th accumulation; and obtaining a piece of imagedata using one or more pieces of image data obtained by the N count ofthe accumulations, the piece of image data being equivalent to a pieceof image data obtainable by predetermined first convolution processing.

According to this configuration, a broader range of coefficients can behandled by combining sensitivity and exposure time.

An imaging method according to another aspect of the present disclosureis an imaging method in an imaging system. The imaging system includesan imaging optical system that images an image of a subject, an imagingdevice including a plurality of pixel cells, an actuator that changes arelative position of the plurality of pixel cells and the image of thesubject, and control circuitry that controls the imaging device and theactuator. The plurality of pixel cells each have variable sensitivity.The plurality of pixel cells each include a photoelectric converter thatconverts light of the image of the subject into a signal charge, and acharge accumulation region that accumulates the signal charge obtainedat the photoelectric converter. The imaging method includes: performing,during one exposure period, changing the relative position from a firstposition to a second position that is different from the first position;changing the sensitivity of each of the plurality of pixel cells from afirst sensitivity to a second sensitivity that is different from thefirst sensitivity; and causing a third signal charge obtained at thephotoelectric converter to be accumulated at the charge accumulationregion, in each of the plurality of pixel cells.

According to this configuration, the imaging method can obtain a pieceof image data where signal charges obtained at different relativepositions and different sensitivities have been accumulated, whilechanging the relative position between the plurality of pixel cells andthe image of the subject, and changing the sensitivity of the pluralityof pixel cells. Accordingly, at least part of convolution computationprocessing can be performed at the imaging device, so the amount ofcomputation of convolution processing can be reduced.

The relative position may be changed from the first position to thesecond position in a continuous manner. According to this configuration,the imaging method can easily realize changing of the relative positionbetween the plurality of pixel cells and the image of the subject, andincrease in shooting time due to this changing can be suppressed.

The sensitivity of each of the plurality of pixel cells may be changedfrom the first sensitivity to the second sensitivity in a continuousmanner. Accordingly, the imaging method can improve replicability ofconvolution computation.

The following is a description of the imaging system and imaging methodaccording to the present disclosure, with reference to the drawings.Note that the embodiments described below are all general or specificexamples. Accordingly, values, shapes, materials, components, layout andconnection state of components, steps, the order of steps, and so forthillustrated in the following embodiments, are only exemplary, and arenot intended to restrict the present disclosure. Various aspectsdescribed in the present embodiment may be combined with each other tothe extent that there is no conflict. Components in the followingembodiments which are not included in the independent Claim indicatingthe most general concept are described as optional components.Components having substantially the same functions may be denoted bycommon reference numerals, and description thereof may be omitted.

First Embodiment

First, convolution processing will be explained. A filter used inconvolution processing has a coefficient group laid outtwo-dimensionally as x×y, as a plurality of elements. The x and y hereare each integers that are 2 or greater. A reference position in thetwo-dimensional layout will be referred to as the center of the filter.In a case of a filter where the number of rows and the number of columnsare both odd, the middle of the layout is often used as the center ofthe filter. FIG. 1 illustrates an example of a filter including 3×3coefficients.

FIG. 2 illustrates an example of an original image. FIG. 3 illustratesnumerical value data where luminance and so forth of the original imagein FIG. 2 has been quantified. FIG. 4 illustrates numerical value dataafter convolution processing has been performed using the filterillustrated in FIG. 1. FIG. 5 illustrates an image corresponding to thenumerical value data in FIG. 4.

Note that in numerical value data corresponding to an image with a pixelcount of N, the filter will run over the edge of the image forconvolution at pixels situated at the edge portions of the display(e.g., pixels in the far left column, etc.). In this case, convolutionis performed virtually assuming that pixels of numerical value 0 existat the portions running over the edge. In the image followingconvolution processing in FIG. 5, pixels where the numerical value isnegative are represented as being black.

Convolution is performed according to the following flow. FIG. 6illustrates the flow of convolution processing with regard to oneselected pixel.

First, a pixel of interest to which the filter is to be applied isselected from yet-to-be selected pixels in the numerical value data ofthe original image laid out two-dimensionally. Next, the center of thefilter is placed over the position of the pixel of interest. Theproducts of numerical value data and filter coefficient values are thencalculated for each overlaid position. The sum of all calculatedproducts is calculated next.

The calculated sum is then stored as the numerical value of the pixel ofinterest after the convolution processing. Determination is maderegarding whether or not all pixels have been processed. If processingof all pixels is complete, the processing ends, and if not completed,the next pixel is selected as the pixel of interest, and theabove-described series of processing is performed on that pixel ofinterest.

In a case of performing convolution processing using a filter having anelement count of M on numerical value data of an image having a pixelcount of N, basically N×M multiplications and M additions need to becomputed. There is a problem that the amount of computations is great ina case where all of the convolution processing is to be realized bysoftware.

The configuration of an imaging system 100 according to the presentembodiment will be described below. FIG. 7 is a conceptual diagramillustrating the configuration of the imaging system 100 according tothe present embodiment. FIG. 7 only illustrates elements necessary fordescription of the present disclosure, and other elements are omitted.Further, the actual shapes, scale, and so forth of the elements are notgiven consideration.

As illustrated in FIG. 7, the imaging system 100 is provided with animaging device 101 whose sensitivity can be electrically changed, animaging optical system 102, a position setting unit 103, a sensitivitysetting circuit 104, a position setting circuit 105, a synchronizationcircuit 106, a mechanical shutter 107, a computing circuit 108, and astorage region 109.

The imaging optical system 102 images an image of a subject on theimaging device 101. The imaging device 101 includes a plurality of pixelcells 10 laid out two-dimensional in the row direction and columndirection, as illustrated in FIG. 8.

The sensitivity setting circuit 104 sets the sensitivity of each of theplurality of pixel cells 10 based on a control signal 111 (also referredto as “second control signal”). The position setting circuit 105 setsthe relative positions of the plurality of pixel cells 10 and thesubject image, based on a control signal 112 (also referred to as “firstcontrol signal”). The synchronization circuit 106 synchronizes thecontrol signal 111 and control signal 112. For example, thesynchronization circuit 106 generates control signals 111 and controlsignals 112 that are synchronized with each other.

FIG. 8 illustrates an exemplary circuit configuration of the imagingdevice 101 according to the present disclosure. The imaging device 101illustrated in FIG. 8 has a pixel array PA including the plurality ofpixel cells 10 arrayed two-dimensionally. FIG. 8 schematicallyillustrates an example where pixel cells 10 are laid out in a two-rowtwo-column matrix. It is needless to say that the number and layout ofpixel cells 10 in the imaging device 101 is not restricted to theexample illustrated in FIG. 8.

The pixel cell 10 includes a photoelectric converter 13 and a signaldetection circuit 14. The photoelectric converter 13 has two opposingelectrodes, and a photoelectric conversion layer interposed betweenthese two electrodes, and generates signal charges upon receivingincident light, which will be described later with reference to thedrawings. One entire photoelectric converter 13 does not have to be anindependent element for each pixel cell 10, and part of a photoelectricconverter 13 may extend over multiple pixel cells 10, for example. Inother words, part of one photoelectric converter 13 may be integrallyformed with part of another photoelectric converter 13. In the case ofthe present embodiment, the incident-side electrode and photoelectricconversion layer extend over part or all of the pixel cell 10.

The signal detection circuit 14 is a circuit that detects signalsgenerated by the photoelectric converter 13. In this example, the signaldetection circuit 14 has a signal detecting transistor 24 and an addresstransistor 26. The signal detecting transistor 24 and address transistor26 typically are field effect transistors (FET), the signal detectingtransistor 24 and address transistor 26 being exemplified here as beingN-channel metal-oxide semiconductor (MOS) devices.

The control terminal (gate in this case) of the signal detectingtransistor 24 has an electrical connection with the photoelectricconverter 13, as schematically illustrated in FIG. 8. Signal charges(holes or electrons) generated at the photoelectric converter 13 areaccumulated at a charge accumulation region (also referred to as“floating diffusion node”) 41 between the gate of the signal detectingtransistor 24 and the photoelectric converter 13. The structure of thephotoelectric converter 13 will be described in detail later.

The photoelectric converter 13 of each pixel cell 10 further hasconnection with a sensitivity control line 42 in the configurationillustrated in FIG. 8. The sensitivity control line 42 is connected to avoltage supply circuit 32. This voltage supply circuit 32 is a circuitconfigured to be capable of supplying at least three types of voltage,which are a first voltage, second voltage, and third voltage, to thephotoelectric converter 13. The voltage supply circuit 32 suppliespredetermined voltage to the photoelectric converter 13 via thesensitivity control line 42 when the imaging device 101 is operating.The voltage supply circuit 32 is not restricted to a particular powersource circuit, and may be a circuit that generates a predeterminedvoltage, or may be a circuit that converts voltage supplied from anotherpower source into a predetermined voltage. Starting and endingaccumulation of signal charges from the photoelectric converter 13 tothe charge accumulation region 41 is controlled by switching of thevoltage supplied from the voltage supply circuit 32 to the photoelectricconverter 13. The voltage is switched between a plurality of voltagesthat are different from each other, which will be described in detaillater. In other words, electronic shutter operations are executed in theembodiments of the present disclosure by switching of the voltagesupplied from the voltage supply circuit 32 to the photoelectricconverter 13. The sensitivity of the photoelectric converter 13 can alsobe switched, by switch of the voltage supplied from the voltage supplycircuit 32 to the photoelectric converter 13. The voltage supply circuit32 is an example of a voltage applying circuit. An example of operationsof the imaging device 101 will be described later.

The pixel cells 10 each have a connection with a power source line 40that supplies power source voltage VDD. An input terminal (typicallydrain) of the signal detecting transistor 24 is connected to the powersource line 40, as illustrated in FIG. 8. The signal detectingtransistor 24 outputs voltage corresponding to the signal chargegenerated by the photoelectric converter 13 as signal voltage, by thepower source line 40 functioning as a source follower power source.

The input terminal (e.g., the drain) of the address transistor 26 isconnected to the output terminal (e.g., the source) of the signaldetecting transistor 24. The output terminal (e.g., the source) of theaddress transistor 26 is connected to one of a plurality of verticalsignal lines 47 arrayed for each column of the pixel array PA. Thecontrol terminal (e.g., the gate) of the address transistor 26 isconnected to an address control line 46, and controlling the potentialof the address control line 46 enables the output of the signaldetecting transistor 24 to be selectively read out to a correspondingvertical signal line 47.

In the example illustrated in FIG. 8, the address control line 46 isconnected to a vertical scanning circuit (also referred to as “rowscanning circuit”) 36. The vertical scanning circuit 36 selects one ormore rows of pixel cells 10 by applying predetermined voltage to thecorresponding address control lines 46. Accordingly, readout of signalsof the selected pixel cells 10 is performed.

The vertical signal line 47 is a main signal line that transmits pixelsignals from the pixel array PA to peripheral circuits. A column signalprocessing circuit (also referred to as “row signal accumulatingcircuit”) 37 is connected to the vertical signal line 47. The columnsignal processing circuit 37 performs noise suppressing signalprocessing (such as correlated double sampling), analog-to-digitalconversion (AD conversion), and so forth. A column signal processingcircuit 37 is provided corresponding to each column of the pixel cells10 in the pixel array PA, as illustrated in FIG. 8. A horizontal signalreadout circuit (also referred to as “column scanning circuit”) 38 isconnected to these column signal processing circuits 37. The horizontalsignal readout circuit 38 sequentially reads signals from the columnsignal processing circuits 37 and output signals to a horizontal commonsignal line 49.

In the configuration exemplified in FIG. 8, the pixel cells 10 each havea reset transistor 28. The reset transistor 28 is a field effecttransistor in the same way as the signal detecting transistor 24 andaddress transistor 26, for example. In the following description, anexample where an N-channel MOS device has been applied to the resettransistor will be described, unless particularly stated otherwise. Thereset transistor 28 is connected between a reset voltage line 44 thatsupplies a reset voltage Vr, and the charge accumulation region 41, asillustrated in FIG. 8. The control terminal of the reset transistor 28(e.g., the gate) is connected to a reset control line 48, and thepotential of the charge accumulation region 41 can be reset to the resetvoltage Vr by controlling the potential of the reset control line 48. Inthis example, the reset control line 48 is connected to the verticalscanning circuit 36. Accordingly, one or more row of the plurality ofpixel cells 10 can be reset by the vertical scanning circuit 36 applyinga predetermined voltage to the corresponding reset control lines 48.

In this example, the reset voltage line 44 that supplies the resetvoltage Vr to the reset transistors 28 is connected to a reset voltagesupply circuit 34 (hereinafter referred to as “reset voltage source34”). It is sufficient for the reset voltage source 34 to have aconfiguration capable of supplying the predetermined reset voltage Vr tothe reset voltage line 44 when the imaging device 101 is operating, andis not restricted to a particular power supply circuit, in the same wayas with the voltage supply circuit 32 described above. The voltagesupply circuit 32 and reset voltage source 34 may each be part of asingle voltage supply circuit, or may be individually independentvoltage supply circuits. Note that one or both of the voltage supplycircuit 32 and reset voltage source 34 may be part of the verticalscanning circuit 36. Alternatively, sensitivity control voltage from thevoltage supply circuit 32 and/or reset voltage Vr from the reset voltagesource 34 may be supplied to the pixel cells 10 via the verticalscanning circuit 36.

The power source voltage VDD of the signal detection circuit 14 may beused as the reset voltage Vr. In this case, a voltage supply circuit(omitted from illustration in FIG. 8) that supplies power source voltageto each pixel cell 10 and the reset voltage source 34 may becommonalized. Further, the power source line 40 and reset voltage line44 can be commonalized, so the wiring of the pixel array PA can besimplified. Note however, that using different voltages from each otherfor the reset voltage Vr and the power source voltage VDD of the signaldetection circuit 14 enables more flexible control of the imaging device101.

FIG. 9 schematically illustrates an exemplary device structure of thepixel cell 10. In the configuration exemplified in FIG. 9, theabove-described signal detecting transistor 24, address transistor 26,and reset transistor 28 are formed on a semiconductor substrate 20. Thesemiconductor substrate 20 is not restricted to a substrate that isentirely of a semiconductor. The semiconductor substrate 20 may be aninsulating substrate where a semiconductor layer is provided on a sidewhere a photosensitive region is formed. Description will be made hereregarding an example of using a P-type silicon (Si) substrate as thesemiconductor substrate 20.

The semiconductor substrate 20 has impurity regions (N-type regionshere) 26 s, 24 s, 24 d, 28 d, and 28 s, and element isolation regions 20t for electrical isolation among the pixel cells 10. The elementisolation regions 20 t are also formed between impurity region 24 d andimpurity region 28 d as well. The element isolation regions 20 t areformed by acceptor ion injection under predetermined injectionconditions, for example.

The impurity regions 26 s, 24 s, 24 d, 28 d, and 28 s typically arediffusion layers formed in the semiconductor substrate 20. The signaldetecting transistor 24 includes impurity regions 24 s and 24 d, and agate electrode 24 g (typically a polysilicon electrode), asschematically illustrated in FIG. 9. The impurity regions 24 s and 24 drespectively serve as the source region and drain region of the signaldetecting transistor 24, for example. The channel region of the signaldetecting transistor 24 is formed between the impurity regions 24 s and24 d.

In the same way, the address transistor 26 includes impurity regions 26s and 24 s, and a gate electrode 26 g (typically a polysiliconelectrode) connected to the address control line 46 (see FIG. 8). Thesignal detecting transistor 24 and address transistor 26 areelectrically connected to each other in this example, by sharing theimpurity region 24 s. The impurity region 26 s functions as a sourceregion, for example, of the address transistor 26. The impurity region26 s has a connection with the vertical signal line 47 (see FIG. 8) thatis omitted from illustration in FIG. 9.

The reset transistor 28 includes impurity regions 28 d and 28 s, and agate electrode 28 g (typically a polysilicon electrode) connected to thereset control line 48 (see FIG. 8). The impurity region 28 s functionsas a source region, for example, of the reset transistor 28. Theimpurity region 28 s has a connection with the reset voltage line 44(see FIG. 8) that is omitted from illustration in FIG. 9.

An inter-layer insulating layer 50 (typically a silicon dioxide layer)is disposed on the semiconductor substrate 20, so as to cover the signaldetecting transistor 24, address transistor 26, and reset transistor 28.A wiring layer 56 may be disposed within the inter-layer insulatinglayer 50, as illustrated in FIG. 9. The wiring layer 56 is typicallyformed of metal such as copper or the like, and may partially includewiring such as the above-described vertical signal line 47, for example.The number of insulating layers in the inter-layer insulating layer 50,and the number of layers included in the wiring layer 56 disposed in theinter-layer insulating layer 50, can be optionally set, and are notrestricted to the example illustrated in FIG. 9.

The above-described photoelectric converter 13 is disposed on theinter-layer insulating layer 50. In other words, the plurality of pixelcells 10 making up the pixel array PA (see FIG. 8) are formed on thesemiconductor substrate 20 in the embodiments of the present disclosure.The plurality of pixel cells 10 arrayed two-dimensionally on thesemiconductor substrate 20 make up a photosensitive region (i.e., apixel region). The distance between the centers of two adjacent pixelcells 10 (i.e., pixel pitch) is around 2 μm, for example.

The photoelectric converter 13 includes a pixel electrode 11, anopposing electrode 12, and a photoelectric conversion layer 15 disposedtherebetween. In this example, the opposing electrode 12 andphotoelectric conversion layer 15 are disposed extending over theplurality of pixel cells 10. On the other hand, the pixel electrode 11is provided for each pixel cell 10, and is spatially separated from thepixel electrodes 11 of the adjacent other pixel cells 10, thereby beingelectrically isolated from the pixel electrodes 11 of the other pixelcells 10.

The opposing electrode 12 typically is a transparent electrode formed ofa transparent electroconductive material. The opposing electrode 12 isdisposed on the side of the photoelectric conversion layer 15, the sidereceiving incident light. Accordingly, light that has been transmittedthrough the opposing electrode 12 enters the photoelectric conversionlayer 15. Note that the light detected by the imaging device 101 is notrestricted to light within the wavelength range of visible light (e.g.,380 nm to 780 nm). The term “transparent” in the present specificationmeans to transmit at least part of a wavelength range that is to bedetected. Electromagnetic waves as a whole, including infrared rays andultraviolet rays, are expressed as “light” in the present specification,for the sake of convenience. Transparent conducting oxides (TCO) such asindium tin oxide (ITO), indium zinc oxide (IZO), aluminum-doped zincoxide (AZO), fluoride-doped tin oxide (FTO), stannic oxide (SnO₂),titanium dioxide (TiO₂), zinc peroxide (ZnO₂), and so forth, can be usedfor the opposing electrode 12, for example.

The photoelectric conversion layer 15 receives incident light andgenerates a hole-electron pair. In the present embodiment, thephotoelectric conversion layer 15 is formed of an organic material.Specific examples of materials making up the photoelectric conversionlayer 15 will be described later.

The opposing electrode 12 has a connection with the sensitivity controlline 42 that is connected to the voltage supply circuit 32, as describedearlier with reference to FIG. 8. The opposing electrode 12 here isformed across the plurality of pixel cells 10. Accordingly, sensitivitycontrol voltage of a predetermined magnitude can be applied to theplurality of pixel cells 10 at once from the voltage supply circuit 32via the sensitivity control line 42. Note that the opposing electrodes12 may be provided separated for each pixel cell 10, as long as thesensitivity control voltage of a predetermined magnitude can be appliedfrom the voltage supply circuit 32. In the same way, the photoelectricconversion layer 15 may be provided separated for each pixel cell 10.

Controlling the potential of the opposing electrode 12 as to the pixelelectrode 11 enables one of the hole and electron of the hole-electronpair generated in the photoelectric conversion layer 15 by photoelectricconversion to be collected by the pixel electrode 11. For example, in acase of using holes as signal charges, holes can be selectivelycollected at the pixel electrode 11 by setting the potential of theopposing electrode 12 to be higher than the pixel electrode 11. A caseof using holes as signal charges will be exemplified below. Of course,electrons can be used as signal charges.

The pixel electrode 11 facing the opposing electrode 12 collects one ofpositive and negative charges generated by photoelectric conversion atthe photoelectric conversion layer 15, by an appropriate bias voltagebeing provided across the opposing electrode 12 and pixel electrode 11.The pixel electrode 11 is formed of metal such as aluminum, copper, orso forth, metal nitride, polysilicon that has been impartedelectroconductivity by being doped with an impurity or the like.

The pixel electrode 11 may be a light-shielding electrode. For example,sufficient light-shielding properties can be realized by forming atantalum nitride (TaN) electrode having a thickness of 101 nm, as thepixel electrode 11. Forming the pixel electrode 11 as a light-shieldingelectrode enables incident light that has passed through thephotoelectric conversion layer 15 to be suppressed from entering thechannel region or impurity region of transistors (in this example, atleast one of the signal detecting transistor 24, address transistor 26,and reset transistor 28) formed on the semiconductor substrate 20. Alight-shielding film may also be formed within the inter-layerinsulating layer 50 using the above-described wiring layer 56.Suppressing incident light to the channel region of transistors formedon the semiconductor substrate 20 enables shifting of transistorproperties (e.g., change in threshold voltage) and so forth to besuppressed. Suppressing incident light to the impurity region formed onthe semiconductor substrate 20 enables noise due to unintendedphotoelectric conversion from occurring at the impurity region. This,suppression of incident light to the semiconductor substrate 20contributes in improved reliability of the imaging device 101.

The pixel electrode 11 is connected to the gate electrode 24 g of thesignal detecting transistor 24 via a plug 52, wiring 53, and a contactplug 54, as schematically illustrated in FIG. 9. In other words, thegate of the signal detecting transistor 24 has electrical connectionwith the pixel electrode 11. The plug 52 and wiring 53 are formed ofmetal such as copper or the like, for example. The plug 52, wiring 53,and contact plug 54 make up at least part of the charge accumulationregion 41 (see FIG. 8) between the signal detecting transistor 24 andthe photoelectric converter 13. The wiring 53 may be part of the wiringlayer 56. The pixel electrode 11 is also connected to the impurityregion 28 d via the plug 52, wiring 53, and a contact plug 55. In theconfiguration exemplified in FIG. 9, the gate electrode 24 g of thesignal detecting transistor 24, the plug 52, wiring 53, contact plugs 54and 55, and the impurity region 28 d that is one of the source regionand drain region of the reset transistor 28, function as the chargeaccumulation region 41 that accumulates signal charges collected by thepixel electrode 11.

Due to signal charges collected by the pixel electrode 11, voltagecorresponding to the amount of signal charges accumulated at the chargeaccumulation region 41 is applied to the gate of the signal detectingtransistor 24. The signal detecting transistor 24 amplifies thisvoltage. The voltage amplified by the signal detecting transistor 24 isselectively read out via the address transistor 26, as signal voltage.

As described above, the imaging device 101 has the plurality of pixelcells 10 two-dimensionally laid out equidistantly in the row directionand column direction. Note that the pitch of the pixels in the rowdirection and the pitch of the pixels in the column direction do nothave to be the same.

The pixel cell 10 has at least one set of photoelectric converter 13 andcharge accumulation region 41. The photoelectric converter 13 functionsto generate signal charges of an amount proportionate to the intensityof light shining on that region. The charge accumulation region 41functions to accumulate the signal charges generated by thecorresponding photoelectric converter 13.

Note that each pixel cell 10 may have a plurality of sets of thephotoelectric converter 13 and charge accumulation region 41. Forexample, in order to perform color imaging, each pixel may have sets ofthe photoelectric converter 13 and charge accumulation region 41 toimage each of red component, green component, and blue component. As aseparate example, each pixel may have a set of the photoelectricconverter 13 and charge accumulation region 41 for high-sensitivityimaging, and a set of the photoelectric converter 13 and chargeaccumulation region 41 for low-sensitivity imaging. In this case,imaging processing described later may be performed by the set of thephotoelectric converter 13 and charge accumulation region 41 forhigh-sensitivity imaging. Imaging processing described later may beperformed by the set of the photoelectric converter 13 and chargeaccumulation region 41 for low-sensitivity imaging. The chargeaccumulation region 41 usually is integrated within the imaging device101, but may be externally disposed.

At least one charge accumulation region 41 exists for each photoelectricconverter 13. However, a configuration may be made where a plurality ofcharge accumulation regions 41 exist for one photoelectric converter 13,and the destination of accumulating signal charges can be changed. Inthis case, image processing described later may be performed using anyone of the plurality of charge accumulation regions 41.

The ratio of the amount of signal charges generated as to the lightintensity by which the photoelectric converter 13 is irradiated iscalled quantum efficiency. The ratio of the amount of signal chargesaccumulated in the charge accumulation region 41 as to the lightintensity by which the photoelectric converter 13 is irradiated iscalled sensitivity. In a case where all signal charges generated at thephotoelectric converter 13 are accumulated in the charge accumulationregion 41, the quantum efficiency and sensitivity are in a proportionaterelation. However, the two may be in a relation other thanproportionate, in a case of a mechanism where part or all of signalcharges generated at the photoelectric converter 13 are not accumulatedat the charge accumulation region 41. For example, in a case of having amechanism where the signal charges generated at the photoelectricconverter 13 are discarded, the sensitivity may be 0 even though thequantum efficiency is finite. The act of accumulating signal chargesthat is generated by the photoelectric converter 13 irradiated by light,in the charge accumulation region 41, is referred to as exposure.

The act including a first exposure, a temporary stop of exposure, and asecond exposure is referred to as multiple exposure. In the multipleexposure, after the first signal charge is accumulated in the chargeaccumulation region 41 by the first exposure, the exposure istemporarily stopped with the first signal charge held at the chargeaccumulation region 41. Thereafter, accumulation of a second signalcharge generated by the second exposure is started in the chargeaccumulation region 41 in addition to the first signal charge. Stoppingexposure can be realized by stopping light irradiation, or settingsensitivity to 0. Setting sensitivity to 0 can be realized by settingquantum efficiency to 0, or by not accumulating signal charges generatedat the photoelectric converter 13 in the charge accumulation region 41.Note that multiple exposure may include third and subsequent exposures.

The imaging device 101 has functions to change its sensitivity topredetermined values according to setting by the sensitivity settingcircuit 104. The sensitivity set in the imaging device 101 may bediscrete values. Alternatively, the sensitivity set in the imagingdevice 101 may be, depending on the case, continuous values. In a caseof replicating convolution processing by software according to aconventional calculator, the former is desirable. The latter can performprocessing similar to the convolution processing by software accordingto a conventional calculator.

In a case of replicating convolution processing by software, the numberof types of sensitivity that can be set to the imaging device 101basically needs to be no less than the number of types of absolutevalues of element values in the filter for performing convolution. Forexample, in the case of the filter illustrated in FIG. 1, the absolutevalues of element values in the filter is the three types of {0, 1, 2}.In this case, the sensitivity of the imaging device 101 needs to be ableto set to {sensitivity 0, standard sensitivity, double standardsensitivity}. The standard sensitivity here means sensitivity that isused as a reference for performing convolution imaging one time, and adifferent sensitivity may be used as the reference sensitivity indifferent convolution imaging. For example, the reference sensitivitymay be changed in accordance with the brightness of the subject, theimaging speed that is necessary, or the like. Of the sensitivitiesnecessary for the imaging device 101, the sensitivity 0 can besubstituted by a later-described shutter function, and accordingly doesnot have to be able to be set.

The imaging device 101 may have negative sensitivity. In a case that theimaging device 101 is set to a negative sensitivity, the imaging device101 has a function to remove signal charge from the charge accumulationregion 41 in proportion to the intensity of light by which thephotoelectric converter 13 is irradiated. Note however, that theaforementioned negative sensitivity is not indispensable in the presentdisclosure.

The charge accumulation region 41 has a function of continuouslyaccumulating signal charges regardless of change of the sensitivity ofthe imaging device 101. That is to say, multiple exposure can beperformed at different sensitivities. Alternatively, the chargeaccumulation region 41 may have a function of continuously accumulatingsignal charges in a case where sensitivity is continuously changedwithout interrupting irradiation of the imaging device 101 by light.

The sensitivity in the imaging device 101 may be changed to the samevalue at all pixels. Alternatively, the sensitivity of each pixel may bechanged to different values, independently from that of other pixels, orthe sensitivity of each block of pixels may be changed to differentvalues, independently from that of other blocks. Alternatively, thesensitivity of a photoelectric converter 13 included in each pixel cell10 may be changed to different values, independently from that of theother converter 13.

The changing of sensitivity desirably is substantially isochronous atall pixel cells 10. That is to say, changing of sensitivity of thepixels by the sensitivity setting circuit 104 desirably is started andcompleted substantially at once.

However, isochronicity is not indispensable, and the start andcompletion time of changing sensitivity may be different for each pixelcell 10. In this case, the imaging system 100 desirably has functionsfor performing exposure only during periods during which all pixel cells10 are at a predetermined sensitivity. That is to say, the imagingsystem 100 desirably has a function to allow the imaging device 101 tobe irradiated by light only in a case where all pixel cells 10 are at apredetermined sensitivity, or a function to allow signal chargesgenerated at the photoelectric converter 13 to be accumulated only in acase where all pixel cells 10 are at a predetermined sensitivity. Afunction where exposure is performed only during a desired period iscalled a shutter mechanism.

Transitioning to a state where exposure can be performed is referred toas opening the shutter. Transitioning to a state where exposure cannotbe performed is referred to as closing the shutter. Shielding theimaging device 101 from irradiation of light is realized by themechanical shutter 107 illustrated in FIG. 7. The mechanism ofcontrolling accumulation of signal charges is referred to as anelectronic shutter.

As for an imaging device 101 that is capable of changing sensitivity inaccordance with the present embodiment, there is the imaging device 101that uses the layered photoelectric converter 13 in which the pixelelectrode 11, photoelectric conversion layer 15, and opposing electrode12 have been layered as illustrated in FIG. 9. There are, among thephotoelectric converters 13, elements where the element sensitivity canbe continuously changed from substantially 0 to a finite value, inaccordance with voltage applied across the pixel electrode 11 and theopposing electrode 12. An example of a characteristics diagram regardingthe relation between applied voltage and sensitivity is illustrated inFIG. 10.

Having the opposing electrode 12 in common among all photoelectricconverters 13 enables the sensitivity of all photoelectric converters 13to be changed at once with isochronicity, simply by changing the voltageof the opposing electrode 12. In this case, the sensitivity settingcircuit 104 has voltage setting functions for the opposing electrode 12.The sensitivity setting circuit 104 controls the voltage supply circuit32 to supply sensitivity control voltage to the photoelectric converter13 via the sensitivity control line 42 and opposing electrode 12.

The opposing electrode 12 may have a plurality of sections, with voltagebeing able to be changed for each section. Combining an imaging device101 configured in this way with an imaging optical system such as a flyeye lens which images a plurality of images of a subject enablesconvolution imaging to be performed at once for a plurality of filters.

The sensitivity of the layered photoelectric converter can also bechanged by controlling the voltage at the pixel electrode 11 side or athird electrode (omitted from illustration), instead of the opposingelectrode 12. In this case, the pixel electrodes 11 are independent foreach photoelectric converter 13, and accordingly, the sensitivity can bechanged individually for each photoelectric converter 13. Using thethird electrode that is disposed near the pixel electrode 11 and thatcan change the voltage for each photoelectric converter 13 enables thesensitivity to be individually changed for each photoelectric converter13 in the same way.

In the case of the imaging device described above, there is a need forlines to supply voltage to each pixel electrode 11 or each thirdelectrode, to enable sensitivity to be set individually for eachphotoelectric converter 13. There are cases where it is physicallydifficult to extend lines for supplying voltage to the plurality ofpixel electrodes 11 or third electrodes of all photoelectric converters13 to the outside of the imaging device 101. In this case, a so-calledselection transistor is disposed. The lines selected by this selectiontransistor are sequentially switched, whereby the multiple pixelelectrodes 11 or third electrodes are sequentially connected to theoutside. Thus, the voltage of the connected electrodes can be set tooptional values.

Alternatively, photoelectric converters 13 may be grouped into sets. Ineach of the sets, the sensitivity of the photoelectric converters 13included in the set is determined to be set to the same, and thephotoelectric converters 13 included in the set is connected to the sameexternal single line. This enables the number of external signal linesto be reduced.

The imaging system 100 has a signal charge amount measuring instrumentfor measuring the amount of signal charges accumulated in each chargeaccumulation region 41. The signal charge amount measuring instrumentmay be provided to each charge accumulation region 41, or the chargeaccumulation regions 41 share the signal charge amount measuringinstrument and measurement is performed by switching. This signal chargeamount measuring instrument is equivalent to, for example, the signaldetection circuit 14 illustrated in FIG. 8.

The signal charge amount measuring instrument may discharge signalcharges in the charge accumulation region 41 by the measurementoperation of the amount of signal charges. That is to say, destructivereadout may be performed. Alternatively, the signal charge amountmeasuring instrument may save the signal charges in the chargeaccumulation region 41 by the measurement operation of the amount ofsignal charges. That is to say, nondestructive readout may be performed.The imaging system 100 also has functions of eliminating signal chargesaccumulated in the charge accumulation region 41, as necessary. Theimaging system 100 also has functions of measuring the remaining chargeamount after having eliminated signal charges form the chargeaccumulation region 41, as necessary. A specific configuration of theseconfigurations is the configuration described in FIG. 8, for example.

The imaging system 100 has the storage region 109 that stores themeasurement results of the amount of signal charges in each chargeaccumulation region 41 that have been measured by the signal chargeamount measuring instrument, as necessary. In a case where the filterused for convolution has both positive and negative elements, and theimaging device 101 does not have negative sensitivity, the imagingsystem 100 desirably has the storage region 109.

The imaging system 100 has the computing circuit 108 that performscomputing based on the values of the storage region 109, as necessary.The storage region 109 may be provided within this computing circuit108, for example, or within the imaging device 101.

The imaging optical system 102 has a function imaging an image of thesubject on the imaging device 101. The imaging optical system 102 may bea refractive optical system using lenses or the like, or may be areflective optical system using curved mirrors or the like.Alternatively, the imaging optical system 102 may be a combined type ofboth. The imaging optical system 102 may include elements such as adiaphragm, filter, or the like, as necessary.

The mechanical shutter 107 controls whether or not the imaging device101 is to be irradiated by light. However, in a case where thesensitivity of the imaging device 101 can be set to 0, or thesensitivity is not 0 but accumulation of signal charges to the chargeaccumulation region 41 can be stopped, i.e., in a case where the imagingdevice 101 has an electronic shutter function and also the electronicshutter has isochronicity at all pixel cells 10, the imaging system 100does not have to include the mechanical shutter 107.

The position setting unit 103 has functions of changing the relativeposition of the image of the subject and the imaging device 101 to apredetermined position, based on control by the position setting circuit105. The change in relative position that is necessary in the presentembodiment is change where the imaging device 101 moves relatively overa plane where the image of the subject imaged by the imaging opticalsystem 102 exists. That is to say, this change is change in a directionperpendicular to the optical axis of the imaging optical system 102.

The layout distance among the pixel cells 10 is the reference for therelative position. That is to say, the relative position is set using aninteger multiple of the center-to-center distance between two adjacentpixel cells 10 as an increment. In the present embodiment, a position ofthe image formed by the optical system when the image is moved to theright from a reference position by one pixel layout distance in the rowdirection is written as (+1, 0). The reference position is a positionused as a reference for the relative position between the image of thesubject and the imaging device 101. A position of the image when theimage is moved to the left from the reference position by one pixellayout distance in the row direction is written as (−1, 0). A positionof the image when the image is moved upward from the reference positionby one pixel layout distance in the column direction is written as (0,+1). A position of the image when the image is moved downward from thereference position by one pixel layout distance in the column directionis written as (0, −1).

Other positions are also written in the same way. That is to say,position (+2, −1) indicates a position of the image when the image ismoved from the reference position by double of one pixel layout distanceto the right in the row direction and one pixel layout distance downwardin the column direction. This holds true for the others as well.

For example, in order to replicate convolution by any filter having(3×3) elements, the nine types {(−1, −1), (−1, 0), (−1, +1), (0, −1),(0, 0), (0, +1), (+1, −1), (+1, 0), (+1, +1)} of positions are required.FIG. 11 illustrates an example of such relative positions.

In the same way, in order to replicate convolution by any filter having(5×5) elements, the 25 types {(−2, −2), (−2, −1), (−2, 0), (−2, +1),(−2, +2), (1, −2), (−1, −1), (−1, 0), (−1, +1), (−1, +2), (0, −2), (0,−1), (0, 0), (0, +1), (0, +2), (+1, −2), (+1, −1), (+1, 0), (+1, +1),(+1, +2), (+2, −2), (+2, −1), (+2, 0), (+2, +1), (+2, +2)} ofpredetermined positions are required. Note however, that in a case wherean element of 0 is included in the filter, the position corresponding tothat element does not need to be included in the settable predeterminedpositions.

The position setting unit 103 desirably has a function to keep therelative position between the image of the subject and the imagingdevice 101 for a predetermined time after the relative position has beenchanged. However, this storing function is not indispensable, and amechanism may be used where the relative position is continuouslychanged, as in simple harmonic motion. Details of this will be describedin a third embodiment.

The position setting unit 103 is an actuator. For example, the positionsetting unit 103 is a mechanism that physically changes the position ofthe imaging device 101 by a moving part such as a stepping motor,piezoelectric device, or the like. Note that the position setting unit103 may be a mechanism that physically changes the position of theentire imaging optical system 102 or part of the components thereof. Theposition setting unit 103 may be a mechanism that is disposed betweenthe imaging optical system 102 and imaging device 101 and shifts theoptical path, or the like. Examples of mechanisms to shift the opticalpath include a mechanism that changes the angle or position of a mirror,and a mechanism that changes the position or angle of a transparentoptical element that shifts light rays passing through.

The position setting unit 103 can be realized by a mechanicalconfiguration that is almost the same as a mechanism for maintaining therelative distance between the image of the subject and the imagingdevice 101, that has conventionally been used to suppress shaking. Notethat in the present embodiment, in a situation where shaking can beexpected, the position where shaking correction is performed can betaken as the reference position, and operations be performed to move therelative position additionally therefrom.

The following is description of an example of an imaging method by theabove-described imaging system 100, where an image with convolutionprocessing having been performed can be obtained with less calculationamount than conventionally. Note that in the following, change ofsensitivity is isochronous.

The filter has a size of F_x×F_y. The values of the elements of thefilter are written as F(i, j). Note that F_x and F_y each are integersthat are 2 or greater, i is integers from 1 to F_x, and j is integersfrom 1 to F_y.

For example, F(1, 1) is the value of an element at the upper left cornerof the filter, and F(F_x, F_y) is the value of an element at the lowerright corner. In the example in FIG. 1, this is F(1, 1)=1, F(1, 2)=0,F(1, 3)=−1, . . . , F(3, 3)=−1. The i, j that give the center of thefilter are i=C_x, j=C_y, respectively. In the example in FIG. 1, this isC_x=2, C_y=2.

FIG. 12 is a flowchart illustrating schematics of imaging processingaccording to the present embodiment. First, the imaging system 100 takesa first image corresponding to elements having positive values in thefilter (S101).

Next, the imaging system 100 takes a second image corresponding toelements having negative values in the filter (S102). Finally, thecomputing circuit 108 subtracts the second image from the first image,thereby yielding an image equivalent to an image after convolutionprocessing.

FIG. 13 is a flowchart illustrating step S101 in detail. First, theimaging system 100 stops exposure, or confirms that exposure is stopped(S111). This may be performed by the mechanical shutter 107 or byelectronic shutter. That is to say, the imaging system 100 mayphysically shield the imaging device 101 from being irradiated by light,or may stop the act of accumulating signal charges generated at thephotoelectric converter 13 in the charge accumulation region 41.Alternatively, the imaging system 100 may set the sensitivity of theimaging device 101 to 0.

This processing is performed to prevent the results of exposure beforethe exposure in step S117, or the effects of an uncontrolled state, fromaffecting the charge accumulation region 41. Accordingly, step S111 canbe omitted if it can be ensured that the results of exposure before theexposure in step S117 or the effects of an uncontrolled state will notaffect the charge accumulation region 41 or will be substantiallynegligible.

Next, the imaging system 100 resets each charge accumulation region 41,or confirms that the charge accumulation region 41 has been reset(S112). This processing is performed to correctly measure the amount ofsignal charges generated by exposure after step S113. Accordingly,resetting can be any act that enables the amount of signal chargesgenerated by exposure after step S113 to be correctly measured, besidessetting the amount of signal charges to 0. For example, processing maybe performed as resetting where the amount of charges alreadyaccumulated in the charge accumulation region 41 before the exposureafter step S117 is detected, and the detected value is recorded and soforth.

Next, the imaging system 100 initializes the position of the elementthat is the object of processing out of the elements in the filter(hereinafter referred to as position of interest) (S113). For example,the imaging system 100 sets p=1, q=1. Here, p and q are variablesprovided for convenience of counting the elements of the filter. Notethat p=1, q=1 indicates the upper left corner of the filter. That is tosay, the upper left corner of the filter is selected as the position ofinterest.

Next, the imaging system 100 determines whether the coefficient valueF(p, q) at the position of interest is positive or not (S114). In a casewhere the value F(p, q) is positive (Yes in S114), the imaging system100 sets the relative position between the image of the subject and theimaging device 101 (S115). Specifically, the position setting circuit105 controls the position setting unit 103 so that the relative positionis (C_x−p, q−C_y).

Next, the sensitivity setting circuit 104 sets the sensitivity of allpixel cells 10 of the imaging device 101 to |(referencesensitivity)×F(p, q)| (S116). Note that in a case where the imagingdevice 101 can realize negative sensitivity, the sensitivity settingcircuit 104 sets the sensitivity of all pixel cells 10 to (referencesensitivity)×F(p, q). Note that in a case where the F(p, q) is the sameas in the previous step S116, the processing of setting sensitivity isperformed by maintaining the sensitivity, for example.

Next, the imaging system 100 starts exposure (S117). Specifically, in acase where the imaging system 100 has the mechanical shutter 107, theimaging system 100 opens the mechanical shutter 107. In a case where thestate is that signal charges generated at the photoelectric converter 13are not accumulated at the charge accumulation region 41, the imagingsystem 100 changes this to a state where the signal charges areaccumulated. That is to say, the imaging system 100 starts exposure atthe relative position set in step S115 and the sensitivity set in stepS116. Thus, accumulation of signal charges to the charge accumulationregion 41 is started.

Next, after a predetermined amount of exposure time has elapsed, theimaging system 100 stops exposure (S118). Specifically, in a case ofhaving detected that the predetermined amount of exposure time haselapsed from the start of exposure, the imaging system 100 sets thesensitivity of the imaging device 101 to 0. Alternatively, the imagingsystem 100 shuts the mechanical shutter or electronic shutter.

In a case where the value F(p, q) is negative or zero in step S114 (Noin S114), or after step S118 has been performed, the imaging system 100selects a next position as the position of interest. Specifically, theimaging system 100 increments the variable p by 1. Thus, the position tothe right side of the immediately-previous position of interest isselected as the position of interest.

Next, the imaging system 100 determines whether the position selectedimmediately before is the end of the row (S120). Specifically, theimaging system 100 determines whether or not p=F_x+1 holds. In a casewhere the position that was selected immediately before is not the endof the row, i.e., p=F_x+1 does not hold (No in S120), the imaging system100 performs the processing of step S114 and thereafter on the positionof interest set in step S119.

On the other hand, in a case where the position that was selectedimmediately before is the end of the row, i.e., p=F_x+1 holds (Yes inS120), the imaging system 100 selects position at the start (left edge)of the next row as the position of interest (S121). Specifically, theimaging system 100 sets p=1, q=q+1.

Next, the imaging system 100 determines whether processing of allpositions has ended (S122). For example, the imaging system 100determines whether the row that had been selected immediately before isthe last row. Specifically, the imaging system 100 determines whetherq=F_y+1 holds or not.

In a case where processing of all position has not ended, i.e., in acase where q=F_y+1 does not hold (No in S122), the imaging system 100performs the processing of step S114 and thereafter on the position ofinterest set in step S121.

In the other hand, in a case where the processing of all positions hasended, i.e., in a case where q=F_y+1 holds (Yes in S122), the imagingsystem 100 measures the signal charge amount (S123). Specifically, theimaging system 100 measures the amount of charges accumulated in thecharge accumulation regions 41 using the signal charge amount measuringinstrument, and stores the obtained results (first image) in the storageregion. According to this processing, multiple exposure of positiveelements in the filter is completed, and a first image is obtained.

Next, processing of multiple exposure corresponding to elements havingnegative values in the filter (S102) will be described. FIG. 14 is aflowchart of this processing. Note that the processing in steps S131through S133 and S135 through S143 are the same as the processing insteps S111 through S113 and S115 through S123 illustrated in FIG. 13, sothe description thereof will be omitted.

In step S134, the imaging system 100 determines whether the filtercoefficient value F(p, q) is negative or not (S134). In a case where thevalue F(p, q) is negative (Yes in S134), the flow transitions to stepS135, and in a case where the value F(p, q) is not negative (No inS134), the flow transitions to step S139.

Multiple exposure of elements having negative values in the filter isperformed by the processing illustrated in FIG. 14, and a second imageis generated. The second image that has been generated is stored in thestorage region as well.

The difference between the first image and second image stored in thestorage region is then calculated in step S103 illustrated in FIG. 12.In a case where the coefficient value at the position of interest isneither positive nor negative, i.e., zero, exposure is not performed.

FIG. 15 is a timing chart illustrating the flow of imaging processing ina case of using the filter illustrated in FIG. 1. Note that FIG. 15 is adiagram schematically illustrating the content of image processing, andmay not necessarily agree with processing carried out with regard toscale of the temporal axis and so forth.

As illustrated in FIG. 15, at time t1 after resetting, the upper left ofthe filter (p=1, q=1) is selected as the position of interest, therelative position is set to 1, −1), and the sensitivity is set to 1(reference sensitivity×1). Exposure is performed from time t1 to time t2in this state.

Next, the mechanical shutter 107 is closed from time t2 to t3, so thestate is a no-exposure state. Changing of the relative position andsensitivity is performed during this time t2 to t3. Note that the timingat which the changing of relative position and changing of sensitivitymay be performed at any timing as within this period.

During time t2 to t3, the middle left of the filter (p=1, q=2) isselected as the position of interest, the relative position is set to(1, 0), and the sensitivity is set to 2 (reference sensitivity×2).Exposure is performed from time t3 to time t4 in this state. That is tosay, in addition to signal charges obtained by the exposure of time t1through t2, signal charges obtained in time t3 through t4 areaccumulated in the charge accumulation region 41.

In the same way, during time t4 to t4, the lower left of the filter(p=1, q=3) is selected as the position of interest, the relativeposition is set to (1, 1), and the sensitivity is set to 1 (referencesensitivity×1). Exposure is performed from time t5 to time t6 in thisstate. That is to say, in addition to signal charges obtained by theexposure of time t1 through t2 and time t3 through t4, signal chargesobtained in time t5 through t6 are accumulated in the chargeaccumulation region 41.

At time t6, signals corresponding to these signal charges (first image)are read out. The second image corresponding to negative elements isgenerated during time t7 through t12 by similar processing, and thesecond image is read out.

Although an example has been illustrated in FIG. 15 where the mechanicalshutter 107 is used, an arrangement may be made where the sensitivity isset to zero instead of closing the mechanical shutter 107, asillustrated in FIG. 16.

As described above, the imaging system 100 according to the presentembodiment can obtain basically the same image as the results of havingperformed convolution by software on an image taken at referencesensitivity. The only computation that is necessary in the aboveprocedures is computation of difference for the count of pixels. That isto say, the amount of computation can be reduced as compared toconventional convolution processing using software. In a case ofconvolution using a filter where the filter elements are only positiveor only negative, even the computation of difference is unnecessary.

Multiple exposure changing the relative position between the image ofthe subject and the imaging device 101 in the above-described proceduresis equivalent to adding multiple pixels obtained as a result of imagingwith the relative position of the image of the subject and the imagingdevice 101 unchanged. Changing sensitivity at the time of multipleexposure is equivalent to multiplying filter coefficients. Accordingly,convolution imaging can be performed by the above-described method.

According to the method of the embodiment described above, the sameimaging can be performed regarding any filter simply by changinginstructions to the sensitivity setting circuit 104 and position settingcircuit 105. Convolution using a plurality of filters regarding aparticular subject can be performed by changing instructions to thesensitivity setting circuit 104 and position setting circuit 105 andrepeating the above procedures.

In the method according to the present embodiment described above, thetime required for imaging is hardly dependent on the number of pixels,except for step S103. Step S103 is only difference computation, so thecalculation load is small. The greater the number of pixels is, and themore filters used for convolution, the greater the amount of necessarytime is reduced, compared to the conventional method.

By forming the size of the image of the subject created by the imagingoptical system 102 to be sufficiently larger than the imaging device101, the image of the subject can constantly cover the entire imagingdevice even if the relative position of the image of the subject and theimaging device 101 is changed. This solves the problem where correctcomputation could not be performed due to the filter extending past theperipheral portions of the image, which has occurred in the conventionalmethod of performing convolution on the results of imaging.

As described above, the imaging system 100 according to the presentembodiment has the imaging optical system 102 that images an image of asubject, the imaging device 101 including the plurality of pixel cells10 laid out two-dimensionally on the row direction and column direction,the position setting circuit 105 that sets the relative position of theplurality of pixel cells 10 and the image of the subject based on firstcontrol signals 112, the sensitivity setting circuit 104 that sets thesensitivity of each of the plurality of pixel cells 10 based on secondcontrol signals 111, and the synchronization circuit 106 thatsynchronizes the first control signals 112 and second control signals111. Each of the plurality of pixel cells 10 has a photoelectricconverter 13 that converts light into signal charges, and a chargeaccumulation region 41 that accumulates signal charges obtained at thephotoelectric converter 13.

The charge accumulation region 41 adds, to signal charges obtained bythe photoelectric converter 13 in a state where the relative position isset to a first relative position and the sensitivity is set to a firstsensitivity, accumulation of signal charges obtained by thephotoelectric converter 13 in a state where the relative position is setto a second relative position that is different from the first relativeposition and the sensitivity is set to a second sensitivity that isdifferent from the first sensitivity.

According to this, the imaging system 100 can obtain an image wheresignal charges obtained at different relative positions and differentsensitivities have been added, while changing the relative position ofthe plurality of pixel cells 10 and the image of the subject, andchanging the sensitivity of the plurality of pixel cells 10.Accordingly, at least part of convolution computation processing can beperformed by the imaging device 101, so the amount of computation forconvolution processing can be reduced.

The position setting circuit 105 sets the relative position inincrements of an integer multiple of the pixel pitch. Accordingly, theimaging system 100 can improve replicability of convolution computation.

The sensitivity setting circuit 104 sets the sensitivity of theplurality of pixel cells 10 at once. Accordingly, the imaging system 100can improve replicability of convolution computation.

The photoelectric converter 13 includes the light-transmitting pixelelectrode 11 connected to the charge accumulation region 41, theopposing electrode 12, and the photoelectric conversion layer 15interposed between the pixel electrode 11 and opposing electrode 12.

The sensitivity of the pixel cell 10 is changed by the voltage appliedacross the pixel electrode 11 and opposing electrode 12 being changed.Accordingly, the imaging system 100 can change the sensitivity of thepixel cell 10 by changing the voltage applied to the photoelectricconverter 13.

As illustrated in FIGS. 13, 15, and so forth, the imaging methodaccording to the present embodiment includes a first relative positionsetting step of setting the relative position of the imaging opticalsystem 102 and imaging device 101 to a first relative position, a firstsensitivity setting step of setting the sensitivity of the plurality ofpixel cells 10 to a first sensitivity, a first accumulating step ofaccumulating a first signal charge obtained at the photoelectricconverter 13 in the charge accumulation region 41 in a state where therelative position is set to the first relative position and thesensitivity is set to the first sensitivity, a second relative positionsetting step of setting the relative position of the imaging opticalsystem 102 and imaging device 101 to a second relative position that isdifferent from the first relative position, a second sensitivity settingstep of setting the sensitivity of the plurality of pixel cells 10 to asecond sensitivity that is different from the first sensitivity, asecond accumulating step of accumulating a second signal charge obtainedat the photoelectric converter 13 in the charge accumulation region 41in addition to the first signals charge, in a state where the relativeposition is set to the second relative position and the sensitivity isset to the second sensitivity.

According to this, in this imaging method, an image can be obtainedwhere signal charges obtained at different relative positions anddifferent sensitivities have been added, while changing the relativeposition of the plurality of pixel cells 10 and the image of thesubject, and the sensitivity of the plurality of pixel cells 10.Accordingly, at least part of convolution computation processing can beperformed by the imaging device 101, so the amount of computation forconvolution processing can be reduced.

In the second relative position setting step, the relative position isshifted from the first relative position to the second relative positionby shifting in increments of an integer multiple of the pixel pitch.Accordingly, the imaging method can improve replicability of convolutioncomputation.

The imaging system 100 further has the mechanical shutter 107 thatshields the imaging device 101 from light. In the second relativeposition setting step, the relative position is changed from the firstrelative position to the second relative position in a state where theimaging device 101 is shielded from light by the mechanical shutter 107,as illustrated in FIG. 15 and so forth. Accordingly, exposure is notperformed during changing of the relative position in this imagingmethod, so replicability of convolution computation can be improved.

Alternatively, in the second relative position setting step, therelative position is changed from the first relative position to thesecond relative position in a state where sensitivity is set to zero, asillustrated in FIG. 16 and so forth. Accordingly, exposure is notperformed during changing of the relative position in this imagingmethod, so replicability of convolution computation can be improved.

As illustrated in FIGS. 12, 13, 15, and so forth, the imaging methodincludes an N count (where N is an integer of 2 or greater) of settingsteps including the first relative position setting step, the secondrelative position setting step, the first sensitivity setting step, andthe second sensitivity setting step, and an N count of accumulatingsteps including the first accumulating step and the second accumulatingstep. In an i'th (where i is an integer of 1 or greater but N orsmaller) setting step, the relative position is set to an i'th relativeposition, and the sensitivity is set to an i'th sensitivity. In an i'thaccumulating step, signal charges obtained at the photoelectricconverter 13 in at the i'th relative position and i'th sensitivity areaccumulated in the charge accumulation region 41. The imaging methodfurther includes a computing step of obtaining an image equivalent to animage after predetermined convolution processing, using one or moreimages obtained by the N count of accumulating steps. The N count ofsetting steps correspond to respective N coefficients out of thecoefficients in convolution processing. An i'th relative position set inthe i'th setting step corresponds to a position of a coefficientcorresponding to the i'th setting step. An i'th sensitivity set in thei'th setting step corresponds to a coefficient corresponding to the i'thsetting step. Accordingly, this imaging method can perform at least partof convolution computation processing by the imaging device 101, so theamount of computation for convolution processing can be reduced.

A first image corresponding to the total value of signals regardingwhich coordinates in convolution processing are positive, and a secondimage corresponding to the total value of signals regarding whichcoordinates in convolution processing are negative, are obtained in theN count of accumulating steps, and an image equivalent to an imagefollowing convolution processing can be obtained in the computing stepby subtracting the second image from the first image, as illustrated inFIG. 12 and so forth. Accordingly, the imaging method can realizeconvolution computation processing including negative coefficients.

The N count of setting steps correspond to respective N coefficients ofwhich the value is not zero, out of the coefficients in convolutionprocessing, as illustrated in FIGS. 13, 15, and so forth. That is tosay, exposure of coefficients where the value is zero is skipped.Accordingly, imaging time can be reduced.

Second Embodiment

An example has been described in the first embodiment where a firstimage corresponding to elements having positive values in a filter and asecond image corresponding to elements having negative values in thefilter are generated, and the difference between the first image and thesecond image is calculated. In a second embodiment, a separate method ofobtaining an image corresponding to a filter will be described. In thepresent embodiment, the imaging system 100 generates a first imagecorresponding to a filter where an offset value have been added to allfilter elements and a second image corresponding to a filter where allelements have been set to the offset value, and calculates thedifference between the first image and second image.

An example of a case of generating images corresponding to two types offilters illustrated in (a) and (b) in FIG. 17 will be described below.First, the largest absolute value of negative coefficients included inthese filters is set to an offset value. In the example of FIG. 17, −2is the negative coefficient having the largest absolute values, so theoffset value is set to 2. Note that the offset value may be a valuelarger than the largest absolute value of negative coefficients.

A filter where the offset value 2 has been added to all elements of thefilter ((c) in FIG. 17) and a filter where all elements have been set tothe offset value 2 ((d) in FIG. 17) are generated for the filterillustrated in (a) in FIG. 17. Performing the same processing as that inthe first embodiment for these two filters yields a first image and asecond image. Subtracting the second image from the first image yieldsthe image corresponding to the filter illustrating in (a) in FIG. 17.Note that there is no need to change sensitivity for the filter whereall elements have been changed to the offset value 2. In this case, theprocessing of setting sensitivity is, for example maintaining thesensitivity at the offset value.

Similarly, filter where the offset value 2 has been added to allelements of the filter ((e) in FIG. 17) and a filter where all elementshave been set to the offset value 2 ((f) in FIG. 17) are set for thefilter illustrated in (b) in FIG. 17. The filters in (d) and (f) in FIG.17 are the same filter here, so the same image can be used for thesecond image used for subtraction. Thus, in the technique according tothe present embodiment, the second image used for subtraction can beused in common for a plurality of filters, so the number of times ofshooting can be reduced.

FIG. 18 is a diagram illustrating an example of filters obtained by thetechnique of the first embodiment. It can be seen from FIG. 18 that twoimages are necessary for each filter ((a) and (b)) with the techniqueaccording to the first embodiment.

On the other hand, exposure at positions where the coefficient is zerocan be skipped by using the technique in the first embodiment, asdescribed earlier. Accordingly, there are cases where the shooting timecan be reduced with regard to filters containing many coefficient zeros,even in cases where a plurality of filter are used.

Further, as a separate method, a single image containing both positiveand negative elements may be generated, instead of shooting two images.Specifically, signal charges corresponding to the amount of exposure canbe removed from the charge accumulation region 41 by applying an inversebias to the photoelectric converter 13. Applying positive and negativevoltages to the photoelectric converters 13 corresponding to thepositive and negative filter elements enables a single image includingboth positive and negative elements to be generated.

As described above, in the imaging method according to the presentembodiment, sensitivity corresponding to values to which an offset valuehas been added to the coefficients of convolution processing andsensitivity corresponding to the offset value are set in an N count ofsetting steps, and a first image and second image are obtained in an Ncount of accumulating steps. The first image is equivalent to an imagefollowing convolution processing using coefficients obtained by addingan offset value to all coefficients in convolution processing. Thesecond image is equivalent to an image after convolution processingusing the offset value as all coefficients. In the computing step, thesecond image is subtracted from the first image, thereby obtaining animage equivalent to an image after convolution processing.

Accordingly, this imaging method can realize convolution computationprocessing including negative coefficients. The second image can be usedin common in a case where multiple images that have been subjected todifferent convolution processing from each other are obtained, soshooting time can be reduced.

Third Embodiment

An example where the relative position of the image of the subject andthe imaging device 101 is moved in increments of an integer multiple ofthe pixel pitch has been described in the first embodiment, but therelative position may be continuously changed. FIGS. 19 and 20 arediagrams illustrating an example of continuous change of relativeposition. The frequency of simple harmonic motion in the row directionand the frequency of simple harmonic motion in the column direction areset to different values, as illustrated in FIGS. 19 and 20. Accordingly,the relative position between the image of the subject and the imagingdevice 101 follows a so-called Lissajous figure, following an orbit thatfills in a rectangle made up of the amplitude of rows and columns. Inthis case, the configuration of the position setting unit 103 cansimplified by performing synchronizing control so that the sensitivityis a desired value when the relative position is at a desired position.This is also advantageous in that temporal deviation is averaged out,since the figure passes through sections corresponding to same filterelements multiple times.

Although FIG. 20 shows sensitivity being changed in stages, anarrangement may be made such as illustrated in FIG. 21, wheresensitivity is changed continuously in conjunction with the relativeposition. FIGS. 20 and 21 each illustrate change in relative positionand sensitivity during one exposure period. This one exposure period maybe divided into multiple.

Thus, in the imaging system 100 according to the present embodiment, theposition setting circuit 105 continuously changes the relative positionduring the exposure period, and the charge accumulation region 41accumulates signal charges obtained at the photoelectric converter 13during the exposure period. Accordingly, the imaging system 100 caneasily realize change in the relative position between the plurality ofpixel cells 10 and image of the subject, and increase in shooting timedue to this changing can be suppressed.

The sensitivity setting circuit 104 continuously changes the sensitivityduring one exposure period. This, the imaging system 100 can improvereplicability of convolution processing. Note that in the presentembodiment, the imaging device 101 has positive and negativesensitivity, so there is no need to calculate the difference between thefirst image and second image. Alternatively, a first image and secondimage may be generated and the difference of these images calculated inthe present embodiment as well, as in the first and second embodiments.That is to say, a first image may be generated in one exposure period,and a second image may be generated in another exposure period.

In a second relative position setting step in the imaging methodaccording to the present embodiment, the relative position is changedcontinuously from a first relative position to a second relativeposition. In a second accumulating step, third signal charges obtainedby the photoelectric converter are added to first signal charges in theperiod during which the relative position is being continuously changed,and accumulated in the charge accumulation region 41. Accordingly, thisimaging method can easily realize changing of the relative positionbetween the plurality of pixel cells 10 and the image of the subject,and can suppress increase in shooting time due to this changing.

In a second sensitivity setting step, the sensitivity is continuouslychanged from a first sensitivity to a second sensitivity, and in asecond accumulating step, the third signal charges obtained at thephotoelectric converter 13 are added to the first signal charges andaccumulated in the charge accumulation region 41, during the periodwhere the relative position is being continuously changed and thesensitivity is being continuously changed. Accordingly, this imagingmethod can improve replicability of convolution computation.

Fourth Embodiment

In a fourth embodiment, description will be made regarding an imagingmethod using an imaging device 101 where a plurality of photoelectricconverters 13 are included in each pixel cell 10 of the imaging device101, and sensitivity can be independently set for each of the pluralityof photoelectric converters 13 included in one pixel cell 10.

The greatest advantage of using the imaging device 101 where theindividual sensitivity can be set for each of the plurality ofphotoelectric converters 13 included in one pixel cell 10 is thatconvolution imaging using a plurality of filters can be performed at thesame time. Alternatively, the imaging time of a filter having positiveand negative elements can be reduced. That is to say, if a sufficientnumber of photoelectric converters 13 is included in one pixel cell 10,a plurality of images corresponding to a plurality of filters can beobtained at the same time.

A case of imaging both the positive and negative components of the twofilters illustrated in (a) and (b) in FIG. 18 at the same time will bedescribed blow. A method of extending the number of filters will beself-evident to those skilled in the art, based on the description ofthe present specification.

FIG. 22 is a diagram illustrating the pixel configuration according tothe present embodiment. The imaging device 101 includes a plurality ofpixel cells 10A, as illustrated in FIG. 22. Each pixel cell 10A includesa plurality of sub-pixels 120. Specifically, in the example illustratedin FIG. 22, the pixel cell 10A includes four sub-pixels 120 (sub-pixelsA through D). Note that the configuration of each of the sub-pixels 120is the same as that of the pixel cell 10 illustrated in FIG. 8, forexample. Note that a sub-pixel 120 includes at least the photoelectricconverter 13, and the charge accumulation region 41 that accumulatessignal charges generated at the photoelectric converter 13. In otherwords, other elements of the components of the pixel cell 10 illustratedin FIG. 8 may be provided for each sub-pixel 120, or may be provided incommon for the plurality of sub-pixels 120.

The present embodiment is configured so that the sensitivity of thesub-pixels A through D can be individually controlled. For example, aconfiguration is made such that different voltage can be applied to thepixel electrodes 11 of the sub-pixels A through D, as described earlier.

All sub-pixels A included in the plurality of pixel cells 10A are set tothe same sensitivity as each other. In the same way, all sub-pixels Bincluded in the plurality of pixel cells 10A are set to the samesensitivity as each other. Also, all sub-pixels C included in theplurality of pixel cells 10A are set to the same sensitivity as eachother. Further, all sub-pixels D included in the plurality of pixelcells 10A are set to the same sensitivity as each other.

The elements of the filters will be respectively written as F1(p, q),F2(p, q). The two filters have the same number of elements. The centerof the filters is p=C_x, q=C_y for both.

In the following, the four photoelectric converters 13 included in thesub-pixels A through D will be written as photoelectric converter 13A,photoelectric converter 13B, photoelectric converter 13C, andphotoelectric converter 13D. The four charge accumulation regions 41included in the sub-pixels A through D will be written as chargeaccumulation region 41A, charge accumulation region 41B, chargeaccumulation region 41C, and charge accumulation region 41D.

The photoelectric converter 13A handles imaging with regard to elementshaving positive values in the first filter ((a) in FIG. 18), thephotoelectric converter 13B handles imaging with regard to elementshaving negative values in the first filter ((a) in FIG. 18), thephotoelectric converter 13C handles imaging with regard to elementshaving positive values in the second filter ((b) in FIG. 18), and thephotoelectric converter 13D handles imaging with regard to elementshaving negative values in the second filter ((b) in FIG. 18). However,it is self-evident that allocations of roles other than these can bemade.

An example of the imaging device 101 where sensitivity can beindividually set for each of the plurality of photoelectric converters13 included in one pixel cell 10A is a configuration where the voltageof the pixel electrode 11 side or third electrode of the layered imagingdevice is controlled, as described above. Imaging devices capable ofsetting individual sensitivity for each photoelectric converter 13, thatuse selection transistors do not readily yield isochronicity at the timeof changing sensitivity. A method of application even in a case whereisochronicity is not obtained will be described in the presentembodiment.

FIG. 23 is a flowchart of an imaging method by the imaging system 100according to the present embodiment. The processing of steps S151through S153 is the same as the processing in steps S111 through S113illustrated in FIG. 13. However, in a case where isochronicity cannot beobtained at the time of changing sensitivity, stopping exposure bysetting sensitivity to 0 is desirably avoided. That is to say, exposureis desirably stopped by the mechanical shutter 107 in the presentembodiment.

In step S154, the imaging system 100 determines whether the filtercoefficient values F1(p, q) and F2(p, q) both are zero. In a case wherethe value F1(p, q) and value F2(p, q) both are zero (Yes in S154), theflow transitions to step S159.

In a case where the value F1(p, q) and value F2(p, q) are not both zero(No in S154), the imaging system 100 sets the relative position of theimage of the subject and the imaging device 101 in the same way as instep S115 (S155). Specifically, the position setting circuit 105controls the position setting unit 103 so that the relative positionwill be (C_x−p, q−C_y).

Next, the sensitivity setting circuit 104 sets the sensitivities of thesub-pixels A through D (S156). Specifically, the sensitivity settingcircuit 104 performs settings of sensitivity in accordance with whetherF1(p, q) and F2(p, q) are positive, negative, or zero.

More specifically, if F1(p, q) is positive, the sensitivity settingcircuit 104 sets the sensitivity of the photoelectric converter 13A to|(reference sensitivity)×F1(p, q)|. If F1(p, q) is negative, thesensitivity setting circuit 104 sets the sensitivity of thephotoelectric converter 13A to 0. If F1(p, q) is 0, the sensitivitysetting circuit 104 sets the sensitivity of the photoelectric converter13A to 0.

If F1(p, q) is positive, the sensitivity setting circuit 104 sets thesensitivity of the photoelectric converter 13B to 0. If F1(p, q) isnegative, the sensitivity setting circuit 104 sets the sensitivity ofthe photoelectric converter 13B to |(reference sensitivity)×F1(p, q)|.If F1(p, q) is 0, the sensitivity setting circuit 104 sets thesensitivity of the photoelectric converter 13B to 0.

If F2(p, q) is positive, the sensitivity setting circuit 104 sets thesensitivity of the photoelectric converter 13C to |(referencesensitivity)×F2(p, q)|. If F2(p, q) is negative, the sensitivity settingcircuit 104 sets the sensitivity of the photoelectric converter 13C to0. If F2(p, q) is 0, the sensitivity setting circuit 104 sets thesensitivity of the photoelectric converter 13C to 0.

If F2(p, q) is positive, the sensitivity setting circuit 104 sets thesensitivity of the photoelectric converter 13D to 0. If F2(p, q) isnegative, the sensitivity setting circuit 104 sets the sensitivity ofthe photoelectric converter 13D to |(reference sensitivity)×F2(p, q)|.If F2(p, q) is 0, the sensitivity setting circuit 104 sets thesensitivity of the photoelectric converter 13D to 0.

Next, the imaging system 100 starts exposure (S157). Note that theprocessing of steps S157 through S163 is the same as the processing ofsteps S117 through S123 illustrated in FIG. 13.

According to the above processing, there are generated a first imagecorresponding to elements in the first filter having positive values, asecond image corresponding to elements in the first filter havingnegative values, a third image corresponding to elements in the secondfilter having positive values, and a fourth image corresponding toelements in the second filter having negative values. The generatedfirst through fourth images are stored in the storage region.

The computing circuit 108 then subtracts the second image from the firstimage, thereby generating an image equivalent to the results ofconvolution using the first filter. The computing circuit 108 alsosubtracts the fourth image from the third image, thereby generating animage equivalent to the results of convolution using the second filter.

As described above, convolution imaging on a plurality of filters can beperformed at once if a plurality of pixel cells 10A each have aplurality of photoelectric converters 13, and the sensitivity of eachphotoelectric converter 13 of one pixel cell 10A can be individuallychanged.

FIG. 24 is a diagram schematically illustrating the flow of an imagingmethod according to the present embodiment. FIG. 24 illustrates anexample of a case of performing convolution shooting corresponding tothe first filter illustrated in (a) in FIG. 18 and the second filterillustrated in (b) in FIG. 18. Note that while there are no positions ofelements of which the value is not 0 in the multiple elements of thefirst filter that overlay positions of elements of which the value isnot 0 in the multiple elements of the second filter, these positions maybe overlaid.

First, the upper left (p=1, q=1) is selected as the position ofinterest, and the relative position is set to (1, −1), as illustrated inFIG. 24. The coefficient at the upper left of the first filter is 1, andthe coefficient at the upper left of the second filter is 0, so thesensitivity of the sub-pixel A corresponding to the element having thepositive value in the first filter is set to 1, and the sensitivities ofthe other sub-pixels B through D are set to zero. Exposure is performedin this state. Next, the upper middle (p=2, q=1) is selected as theposition of interest, and the relative position is set to (0, −1). Thecoefficient at the upper middle of the first filter is 0, and thecoefficient at the upper middle of the second filter is −1, so thesensitivity of the sub-pixel D corresponding to the element having thenegative value in the second filter is set to 1, and the sensitivitiesof the other sub-pixels A through C are set to zero. Exposure is thenperformed in this state.

Processing is thereafter performed in the same way. Note that thecoefficients of the first filter and second filter are both zero at themiddle position (p=2, q=2), so exposure is skipped.

As described above, in the imaging system 100 according to the presentembodiment, each of the plurality of pixel cells 10A includes a firstsub-pixel and a second sub-pixel. The first sub-pixel and secondsub-pixel each include a photoelectric converter 13 and chargeaccumulation region 41. The sensitivity setting circuit 104independently sets the sensitivity of the plurality of first sub-pixelsand the sensitivity of the plurality of second sub-pixels included inthe plurality of pixel cells 10A. Accordingly, the imaging system 100can generated a plurality of images obtained by different convolutioncomputations at the same time.

In the imaging method according to the present embodiment, in an N countof setting steps, sensitivity of a plurality of first sub-pixels andsensitivity of a plurality of second sub-pixels are set to differentvalues. In a computing step, one or more images obtained from theplurality of first sub-pixels in the N count of accumulation steps areused to obtain a first image equivalent to an image after predeterminedfirst convolution processing. Also, in the computing step, one or moreimages obtained from the plurality of second sub-pixels in the N countof accumulation steps are used to obtain a second image equivalent to animage after predetermined second convolution processing that isdifferent from the first convolution processing. Accordingly, thisimaging method can generate a plurality of images obtained by differentconvolution computation at the same time.

Although an imaging apparatus according to the present embodiment hasbeen described, the present disclosure is not restricted to thisembodiment. For example, an example has been described where theexposure time is constant for each exposure in multiple exposure, butthe exposure time may be changed in addition to changing thesensitivity. The amount of signal charges accumulated at the chargeaccumulation region 41 is proportionate to the sensitivity and exposuretime. For example, in a case where the sensitivity is set to double thereference sensitivity, and the exposure time is set to double areference exposure time, the amount of signal charges will be four timesa case where the sensitivity is the reference sensitivity and theexposure time is the reference exposure time. That is to say, in a casewhere the amount of signal charges in a case where the sensitivity isset to the reference sensitivity and the exposure time is set to thereference exposure time corresponds to filter coefficient value 1,exposure can be realized corresponding to filter coefficient value 4 bydoubling the sensitivity from the reference sensitivity and doubling theexposure time from the reference exposure time. Thus, by changing theexposure time in addition to sensitivity, a broader range of filtercoefficients can be handled while suppressing increase in the types ofsensitivity used.

An example has been described where the exposure time is constant foreach exposure in multiple exposure, but the exposure time may be changedfor each exposure. For example, in a case where the range of absolutevalues that filter coefficients can assume is integers in the range of 0to 100, the above-described filter coefficients can be realized by acombination of sensitivity and exposure time, where the sensitivitysetting range is {reference sensitivity, two times referencesensitivity, . . . , 10 times reference sensitivity}, from thecombination of sensitivity and exposure time. That is to say, theproduct of sensitivity and exposure time corresponds to the filtercoefficient value.

In a case of performing the same sensitivity changing at all pixels asdescribed in the first embodiment for example, changing exposure time iseasily applied. The same advantages can be obtained by keeping thesensitivity constant and changing the exposure time instead of changingthe sensitivity. However, it is easier to change the sensitivity foreach pixel than to change the exposure time for each pixel. Accordingly,in a case of performing imaging for convolution with a plurality offilters at the same time, the method of changing sensitivity isdesirable.

Although the configuration illustrated in FIG. 9 has been described asan imaging device 101 of which the sensitivity is variable, but anotherimaging device 101 having variable sensitivity may be used. For example,an example of an imaging device 101 of which the sensitivity is variableis an image-intensifier charge-coupled device (ICCD). An ICCD is adevice that generates electrons by incident light to the imaging device,amplifies the generated electrons at a microchannel plate or the like,and measures the obtained signals at a CCD. Changing the voltage appliedto the microchannel plate enables sensitivity to be changed. Forexample, setting the voltage to 0 enables the sensitivity to besubstantially 0. However, ICCDs require high voltage to drive themicrochannel plate. The structure is also complicated and difficult tominiaturize.

Another example of an imaging device 101 of which the sensitivity isvariable is an imaging device where avalanche photodiodes areintegrated. Changing the bias voltage of the avalanche photodiodesenables the avalanche amplification to be changed and change thesensitivity. However, avalanche photodiodes also require high voltage todrive.

The divisions of functional blocks in the block diagrams are exemplary,and a plurality of functional blocks may be realized as a singlefunctional block, a single functional block may be divided into two ormore elements, and a part of functions may be moved to a differentfunctional block.

For example, although the imaging device 101 and the computing circuit108 are shown as separate blocks in FIG. 7, the computing circuit 108may be included in an integrated circuit (e.g., image sensor) includingthe imaging device 101. Alternatively, the computing circuit 108 may beincluded in a signal processing circuit realized by a microprocessorsuch as a digital signal processor (DSP). The sensitivity settingcircuit 104, position setting circuit 105, and synchronization circuit106 make up control circuitry. That is to say, the sensitivity settingcircuit 104, position setting circuit 105, and synchronization circuit106 are realized by the control circuitry. This control circuitryexecutes the processing described with reference to FIGS. 12 through 21,23, and 24, by controlling the components of the imaging system 100.This control circuitry may be configured of one integrated circuit, ormay be configured of a plurality of circuits. The imaging system 100 mayfurther include memory storing a program for executing the processingdescribed with reference to FIGS. 12 through 21, 23, and 24. The controlcircuitry may read out the program stored in memory, and executeprocessing following the program read out.

The processing units included in the imaging device according to theabove-described embodiments typically are realized as large scaleintegration (LSI) circuits that are integrated circuits. These may beindividually formed into one chip, or part or all may be included in onechip.

Circuit integration is not restricted to LSIs, and dedicated circuits orgeneral-purpose processors may be used to realize the same. A fieldprogrammable gate array (FPGA) which can be programmed aftermanufacturing the LSI, or a reconfigurable processor where circuit cellconnections and settings within the LSI can be reconfigured, may beused.

Although an imaging apparatus according to one or multiple forms havebeen described by way of embodiments, the present disclosure is notrestricted to these embodiments. Modifications conceivable by oneskilled in the art made to the embodiments, and forms constructed bycombining components of different embodiments, without departing fromthe essence of the present disclosure, may also be included in the scopeof one or multiple forms.

The imaging system and imaging method according to the presentdisclosure is capable of obtaining images in which convolution has beenperformed at high speed. Accordingly, the imaging system and imagingmethod according to the present disclosure is useful for deep learningprocessing that requires great amounts of convolution processing.Particularly, the usefulness is high in cases where on-the-fly deeplearning processing is required, such as in self-driving and so forth.

What is claimed is:
 1. An imaging system, comprising: an imaging opticalsystem that images an image of a subject; an imaging device including aplurality of pixel cells; an actuator that changes a relative positionof the plurality of pixel cells and the image of the subject; andcontrol circuitry that controls the imaging device and the actuator,wherein the plurality of pixel cells each have variable sensitivity,wherein the plurality of pixel cells each include a photoelectricconverter that converts light of the image of the subject into a signalcharge, and a charge accumulation region that accumulates the signalcharge obtained at the photoelectric converter, and wherein the controlcircuitry sets the relative position to a first position, and also setsthe sensitivity of each of the plurality of pixel cells to a firstsensitivity, to cause a first signal charge obtained at thephotoelectric converter to be accumulated in the charge accumulationregion in each of the plurality of pixel cells, and sets the relativeposition to a second position that is different from the first position,and also sets the sensitivity of each of the plurality of pixel cells toa second sensitivity that is different from the first sensitivity, tocause a second signal charge obtained at the photoelectric converter tobe accumulated in the charge accumulation region in each of theplurality of pixel cells in addition to the first signal charge.
 2. Theimaging system according to claim 1, wherein the control circuitrysynchronously performs the setting of the relative position to the firstposition, and the setting of the sensitivity of each of the plurality ofpixel cells to the first sensitivity, and synchronously performs thesetting of the relative position to the second position, and the settingof the sensitivity of each of the plurality of pixel cells to the secondsensitivity.
 3. The imaging system according to claim 1, wherein thecontrol circuitry performs the setting to the second position bychanging the relative position from the first position to the secondposition by an integer multiple of a center-to-center distance betweentwo adjacent pixel cells out of the plurality of pixel cells.
 4. Animaging system, comprising: an imaging optical system that images animage of a subject; an imaging device including a plurality of pixelcells; an actuator that changes a relative position of the plurality ofpixel cells and the image of the subject; and control circuitry thatcontrols the imaging device and the actuator, wherein the plurality ofpixel cells each have variable sensitivity, wherein the plurality ofpixel cells each include a photoelectric converter that converts lightof the image of the subject into a signal charge, and a chargeaccumulation region that accumulates the signal charge obtained at thephotoelectric converter, and wherein, during one exposure period, thecontrol circuitry changes the relative position from a first position toa second position that is different from the first position, and changesthe sensitivity of each of the plurality of pixel cells from a firstsensitivity to a second sensitivity that is different from the firstsensitivity, to cause a third signal charge obtained at thephotoelectric converter to be accumulated at the charge accumulationregion, in each of the plurality of pixel cells.
 5. The imaging systemaccording to claim 4, wherein the control circuitry changes the relativeposition from the first position to the second position in a continuousmanner.
 6. The imaging system according to claim 4, wherein the controlcircuitry changes the sensitivity of each of the plurality of pixelcells from the first sensitivity to the second sensitivity in acontinuous manner.
 7. The imaging system according to claim 4, whereinthe control circuitry performs the changing of the relative position andthe changing of the sensitivity of each of the plurality of pixel cellsin a synchronous manner.
 8. The imaging system according to claim 1,wherein the plurality of pixel cells are laid out two-dimensionally in arow direction and a column direction.
 9. The imaging system according toclaim 1, wherein the control circuitry performs the setting of thesensitivity of each of the plurality of pixel cells all at once.
 10. Theimaging system according to claim 1, wherein each of the plurality ofpixel cells includes a first sub-pixel and a second sub-pixel, whereinthe first sub-pixel of each of the plurality of pixel cells includes thephotoelectric converter and the charge accumulation region, wherein thesensitivity of each of the plurality of pixel cells is the sensitivityof the first sub-pixel, wherein the second sub-pixel of each of theplurality of pixel cells includes a second photoelectric converter and asecond charge accumulation region, and wherein the control circuitrysets or changes sensitivity of the second sub-pixel independently fromthe sensitivity of the first sub-pixel in each of the plurality of pixelcells.
 11. The imaging system according to claim 1, wherein thephotoelectric converter of each of the plurality of pixel cellsincludes: a pixel electrode connected to the charge accumulation region;an opposing electrode that transmits light; and a photoelectricconversion layer disposed between the pixel electrode and the opposingelectrode.
 12. The imaging system according to claim 11, wherein theimaging device further includes a voltage applying circuit that appliesvoltage across the pixel electrode and the opposing electrode includedin the photoelectric converter of each of the plurality of pixel cells,and wherein the control circuitry changes the sensitivity of each of theplurality of pixel cells by changing the voltage that the voltageapplying circuit applies.
 13. An imaging method in an imaging system,the imaging system comprising: an imaging optical system that images animage of a subject; an imaging device including a plurality of pixelcells; an actuator that changes a relative position of the plurality ofpixel cells and the image of the subject; and control circuitry thatcontrols the imaging device and the actuator, wherein the plurality ofpixel cells each have variable sensitivity, and the plurality of pixelcells each include: a photoelectric converter that converts light of theimage of the subject into a signal charge; and a charge accumulationregion that accumulates the signal charge obtained at the photoelectricconverter, the imaging method comprising: setting the relative positionto a first position; setting the sensitivity of each of the plurality ofpixel cells to a first sensitivity; causing a first signal chargeobtained at the photoelectric converter to be accumulated in the chargeaccumulation region in each of the plurality of pixel cells in a statewhere the relative position is set to the first position and thesensitivity of each of the plurality of pixel cells set to the firstsensitivity; setting the relative position to a second position that isdifferent from the first position; setting the sensitivity of each ofthe plurality of pixel cells to a second sensitivity that is differentfrom the first sensitivity; and causing a second signal charge obtainedat the photoelectric converter to be accumulated in the chargeaccumulation region in each of the plurality of pixel cells in additionto the first signal charge, in a state where the relative position isset to the second position and the sensitivity of each of the pluralityof pixel cells set to the second sensitivity.
 14. The imaging methodaccording to claim 13, wherein the setting to the second position isperformed by changing the relative position from the first position tothe second position by an integer multiple of a center-to-centerdistance between two adjacent pixel cells out of the plurality of pixelcells.
 15. The imaging method according to claim 13, wherein the imagingsystem further comprises a mechanical shutter that shields imagingdevice from light, and wherein the setting to the second position isperformed by changing the relative position from the first position tothe second position in a state where the imaging device is shielded fromlight by the mechanical shutter.
 16. The imaging method according toclaim 13, wherein the setting to the second position is performed bychanging the relative position from the first position to the secondposition in a state where the sensitivity of each of the plurality ofpixel cells set to zero.
 17. The imaging method according to claim 13,further comprising: performing an N count (where N is an integer of 2 orgreater) of settings of the relative position, including the setting tothe first position and the setting to the second position, in which therelative position is set to an i'th (where i is an integer of 1 to N)position in an i'th setting of the relative position; performing an Ncount of settings of the sensitivity, including the setting to the firstsensitivity and the setting to the second sensitivity, in which thesensitivity of each of the plurality of pixel cells is set to an i'thsensitivity in an i'th setting of the sensitivity; performing an N countof accumulations, including the accumulation of the first signal chargeand the accumulation of the second signal charge, in which the i'thsignal charge obtained at the photoelectric converter is accumulated atthe charge accumulation region at each of the plurality of pixel cellsin a state where the relative position is set to the i'th position andthe sensitivity of each of the plurality of pixel cells is set to thei'th sensitivity at the i'th accumulation; and obtaining a piece ofimage data using one or more pieces of image data obtained by the Ncount of the accumulations, the piece of image data being equivalent toa piece of image data obtainable by predetermined first convolutionprocessing, wherein the i'th position corresponds to a position of ani'th coefficient included in an N count of coefficients out of aplurality of coefficients in the first convolution processing, andwherein the i'th sensitivity corresponds to the value of the i'thcoefficient.
 18. The imaging method according to claim 17, wherein theone or more pieces of image data includes a first piece of image dataand a second piece of image data, wherein the N is an integer of 3 orgreater, wherein the N count of coefficients is made up of an M (whereinM is an integer of 1 or greater but smaller than N) count ofcoefficients having a positive value, and an (N−M) count of coefficientshaving a negative value, wherein the first piece of image datacorresponds to a sum value of signal charges accumulated in the chargeaccumulation region of each of the plurality of pixel cells, by the Mcount of accumulations corresponding to the M count of coefficients,wherein the second piece of image data corresponds to a sum value ofsignal charges accumulated in the charge accumulation region of each ofthe plurality of pixel cells, by an (N−M) count of accumulationscorresponding to the (N−M) count of negative coefficients, and whereinthe piece of image data equivalent to the piece of image data obtainableby the first convolution processing is obtained by subtracting thesecond piece of image data from the first piece of image data.
 19. Theimaging method according to claim 13, further comprising: performing anN count (where N is an integer of 4 or greater) of settings of therelative position, including the setting to the first position and thesetting to the second position, in which the relative position is set toan i'th (where i is an integer of 1 to N) position in an i'th setting ofthe relative position; performing an N count of settings of thesensitivity, including the setting to the first sensitivity and thesetting to the second sensitivity, in which the sensitivity of each ofthe plurality of pixel cells is set to an i'th sensitivity in an i'thsetting of the sensitivity; performing an N count of accumulations,including the accumulation of the first signal charge and theaccumulation of the second signal charge, in which the i'th signalcharge obtained at the photoelectric converter is accumulated at thecharge accumulation region at each of the plurality of pixel cells in astate where the relative position is set to the i'th position and thesensitivity of each of the plurality of pixel cells is set to the i'thsensitivity at the i'th accumulation; performing a setting to asensitivity corresponding to a sum value of an offset value and one of aplurality of coefficients of predetermined first convolution processing,in each of an M (wherein M is an integer of 2 or greater but smallerthan N) counts of settings, out of the N counts of settings of thesensitivity; performing a setting to sensitivity corresponding to theoffset value in each of the (N−M) count of settings, out of the N countsof settings of the sensitivity; obtaining a first piece of image data byan M count of accumulations corresponding to the M count of settings ofthe sensitivity, the first piece of image data being equivalent to apiece of image data obtainable by convolution processing using aplurality of coefficients obtainable by adding an offset value to allcoefficients of the first convolution processing; obtaining a secondpiece of image data by an (N−M) count of accumulations corresponding tothe (N−M) count of settings of the sensitivity, the second piece ofimage data being equivalent to a piece of image data obtainable byconvolution processing using the offset value as all coefficients; andobtaining a piece of image data by subtracting the second piece of imagedata from the first piece of image data, the piece of image data beingequivalent to a piece of image data obtainable by the first convolutionprocessing.
 20. The imaging method according to claim 17, wherein thevalue of the N count of coefficients used for the N count of settings ofthe sensitivity is not
 0. 21. The imaging method according to claim 17,further comprising: obtaining a piece of image data equivalent to apiece of image data obtainable by second convolution processing that isdifferent from the first convolution processing, wherein each of theplurality of pixel cells includes a first sub-pixel and a secondsub-pixel, wherein the first sub-pixel of each of the plurality of pixelcells includes the photoelectric converter and the charge accumulationregion, wherein the sensitivity of each of the plurality of pixel cellsis sensitivity of the first sub-pixel, wherein the second sub-pixel ofeach of the plurality of pixel cells includes a second photoelectricconverter and a second charge accumulation region, wherein, in each ofthe N count of settings of the sensitivity, sensitivity of the secondsub-pixel in each of the plurality of pixel cells is further set,wherein, in at least one of the N count of settings of the sensitivity,a setting value of the sensitivity of the first sub-pixel is differentfrom a setting value of the sensitivity of the second sub-pixel, whereinin each of the N count of accumulations, a signal charge obtained at thesecond photoelectric converter of the second sub-pixel is accumulated inthe second charge accumulation region, in each of the plurality of pixelcells, and wherein the piece of image data equivalent to the piece ofimage data obtainable by the first convolution processing is obtainedusing one or more pieces of image data obtained by the plurality offirst sub-pixels by the N count of accumulations, and the piece of imagedata equivalent to the piece of image data obtainable by the secondconvolution processing is obtained using one or more other imagesobtained by the plurality of second sub-pixels by the N count ofaccumulations.
 22. The imaging method according to claim 13, furthercomprising: performing an N count (where N is an integer of 2 orgreater) of settings of the relative position, including the setting tothe first position and the setting to the second position, in which therelative position is set to an i'th (where i is an integer of 1 to N)position in an i'th setting of the relative position; performing an Ncount of settings of the sensitivity, including the setting to the firstsensitivity and the setting to the second sensitivity, in which thesensitivity of each of the plurality of pixel cells is set to an i'thsensitivity in an i'th setting of the sensitivity, and the exposure timeis set to an i'th exposure time; performing an N count of accumulations,including the accumulation of the first signal charge and theaccumulation of the second signal charge, in which the i'th signalcharge obtained at the photoelectric converter in the i'th exposure timeis accumulated at the charge accumulation region at each of theplurality of pixel cells in a state where the relative position is setto the i'th position and the sensitivity of each of the plurality ofpixel cells is set to the i'th sensitivity at the i'th accumulation; andobtaining a piece of image data using one or more pieces of image dataobtained by the N count of the accumulations, the piece of image databeing equivalent to a piece of image data obtainable by predeterminedfirst convolution processing.
 23. An imaging method in an imagingsystem, the imaging system comprising: an imaging optical system thatimages an image of a subject; an imaging device including a plurality ofpixel cells; an actuator that changes a relative position of theplurality of pixel cells and the image of the subject; and controlcircuitry that controls the imaging device and the actuator, wherein theplurality of pixel cells each have variable sensitivity, and theplurality of pixel cells each include: a photoelectric converter thatconverts light of the image of the subject into a signal charge; and acharge accumulation region that accumulates the signal charge obtainedat the photoelectric converter, the imaging method comprising:performing, during one exposure period, changing the relative positionfrom a first position to a second position that is different from thefirst position; changing the sensitivity of each of the plurality ofpixel cells from a first sensitivity to a second sensitivity that isdifferent from the first sensitivity; and causing a third signal chargeobtained at the photoelectric converter to be accumulated at the chargeaccumulation region, in each of the plurality of pixel cells.
 24. Theimaging method according to claim 23, wherein the relative position ischanged from the first position to the second position in a continuousmanner.
 25. The imaging system according to claim 23, wherein thesensitivity of each of the plurality of pixel cells is changed from thefirst sensitivity to the second sensitivity in a continuous manner.